2023-12-06T03:36:53.932 INFO:root:teuthology version: 0.0.1.dev214+gf2de108 2023-12-06T03:36:53.933 DEBUG:teuthology.run:Teuthology command: teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --owner scheduled_yuriw@teuthology --archive /home/teuthworker/archive/yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7478700 --name yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi --verbose -- /home/teuthworker/archive/yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7478700/orig.config.yaml 2023-12-06T03:36:53.986 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-06T03:36:54.073 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7478700 branch: wip-yuri6-testing-2023-12-05-0753-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7478700' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri6-testing-2023-12-05-0753-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri6-testing-2023-12-05-0753-pacific sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 5202 sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri6-testing-2023-12-05-0753-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b targets: smithi132.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK1U4aUxfPIY9Ie8QujEKrtiRdKt/HGMndDrVZPcL6Lv/9HtaQ8oih0BQ9hW/0fg6yo1+Ob9qAed8A6b96J8GVo= smithi152.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBmZf2ZACUh5sE2dzDfab8lzB22ec/8jLW/QzpPhSqZiwXTDE8pNZZN0xVpKPTSIrK472EVbbw1FYTytwYTp4+M= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: f2de108beb0a6dde7f3e6c96039071828a1f6269 timestamp: 2023-12-05_19:00:15 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.232157 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-12-06T03:36:54.073 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa; will attempt to use it 2023-12-06T03:36:54.074 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks 2023-12-06T03:36:54.074 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-06T03:36:54.076 INFO:teuthology.task.internal:Checking packages... 2023-12-06T03:36:54.101 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'd03fca4f35d43f61b4679936aef7bd2f01cced0b' 2023-12-06T03:36:54.102 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-06T03:36:54.102 INFO:teuthology.packaging:ref: None 2023-12-06T03:36:54.102 INFO:teuthology.packaging:tag: None 2023-12-06T03:36:54.102 INFO:teuthology.packaging:branch: wip-yuri6-testing-2023-12-05-0753-pacific 2023-12-06T03:36:54.102 INFO:teuthology.packaging:sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b 2023-12-06T03:36:54.102 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri6-testing-2023-12-05-0753-pacific 2023-12-06T03:36:54.246 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-335.gd03fca4f 2023-12-06T03:36:54.246 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-06T03:36:54.252 INFO:teuthology.task.internal:no buildpackages task found 2023-12-06T03:36:54.253 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-06T03:36:54.280 INFO:teuthology.task.internal:Saving configuration 2023-12-06T03:36:54.299 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-06T03:36:54.305 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-06T03:36:54.329 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7478700', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-06 03:31:40.837514', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK1U4aUxfPIY9Ie8QujEKrtiRdKt/HGMndDrVZPcL6Lv/9HtaQ8oih0BQ9hW/0fg6yo1+Ob9qAed8A6b96J8GVo='} 2023-12-06T03:36:54.351 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7478700', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-06 03:31:40.836033', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBmZf2ZACUh5sE2dzDfab8lzB22ec/8jLW/QzpPhSqZiwXTDE8pNZZN0xVpKPTSIrK472EVbbw1FYTytwYTp4+M='} 2023-12-06T03:36:54.351 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-06T03:36:54.359 INFO:teuthology.task.internal:roles: ubuntu@smithi132.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-12-06T03:36:54.359 INFO:teuthology.task.internal:roles: ubuntu@smithi152.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-12-06T03:36:54.359 INFO:teuthology.run_tasks:Running task console_log... 2023-12-06T03:36:54.455 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6bcba46a60>, signals=[15]) 2023-12-06T03:36:54.456 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-06T03:36:54.463 INFO:teuthology.task.internal:Opening connections... 2023-12-06T03:36:54.463 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T03:36:54.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T03:36:54.533 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi152.front.sepia.ceph.com 2023-12-06T03:36:54.535 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T03:36:54.644 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-06T03:36:54.650 DEBUG:teuthology.orchestra.run.smithi132:> uname -m 2023-12-06T03:36:54.675 INFO:teuthology.orchestra.run.smithi132.stdout:x86_64 2023-12-06T03:36:54.675 DEBUG:teuthology.orchestra.run.smithi132:> cat /etc/os-release 2023-12-06T03:36:54.735 INFO:teuthology.orchestra.run.smithi132.stdout:NAME="CentOS Stream" 2023-12-06T03:36:54.736 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION="8" 2023-12-06T03:36:54.736 INFO:teuthology.orchestra.run.smithi132.stdout:ID="centos" 2023-12-06T03:36:54.736 INFO:teuthology.orchestra.run.smithi132.stdout:ID_LIKE="rhel fedora" 2023-12-06T03:36:54.736 INFO:teuthology.orchestra.run.smithi132.stdout:VERSION_ID="8" 2023-12-06T03:36:54.736 INFO:teuthology.orchestra.run.smithi132.stdout:PLATFORM_ID="platform:el8" 2023-12-06T03:36:54.736 INFO:teuthology.orchestra.run.smithi132.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-06T03:36:54.736 INFO:teuthology.orchestra.run.smithi132.stdout:ANSI_COLOR="0;31" 2023-12-06T03:36:54.736 INFO:teuthology.orchestra.run.smithi132.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-06T03:36:54.736 INFO:teuthology.orchestra.run.smithi132.stdout:HOME_URL="https://centos.org/" 2023-12-06T03:36:54.736 INFO:teuthology.orchestra.run.smithi132.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-06T03:36:54.736 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-06T03:36:54.736 INFO:teuthology.orchestra.run.smithi132.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-06T03:36:54.737 INFO:teuthology.lock.ops:Updating smithi132.front.sepia.ceph.com on lock server 2023-12-06T03:36:54.761 DEBUG:teuthology.orchestra.run.smithi152:> uname -m 2023-12-06T03:36:54.783 INFO:teuthology.orchestra.run.smithi152.stdout:x86_64 2023-12-06T03:36:54.783 DEBUG:teuthology.orchestra.run.smithi152:> cat /etc/os-release 2023-12-06T03:36:54.845 INFO:teuthology.orchestra.run.smithi152.stdout:NAME="CentOS Stream" 2023-12-06T03:36:54.845 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION="8" 2023-12-06T03:36:54.845 INFO:teuthology.orchestra.run.smithi152.stdout:ID="centos" 2023-12-06T03:36:54.845 INFO:teuthology.orchestra.run.smithi152.stdout:ID_LIKE="rhel fedora" 2023-12-06T03:36:54.845 INFO:teuthology.orchestra.run.smithi152.stdout:VERSION_ID="8" 2023-12-06T03:36:54.845 INFO:teuthology.orchestra.run.smithi152.stdout:PLATFORM_ID="platform:el8" 2023-12-06T03:36:54.845 INFO:teuthology.orchestra.run.smithi152.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-06T03:36:54.845 INFO:teuthology.orchestra.run.smithi152.stdout:ANSI_COLOR="0;31" 2023-12-06T03:36:54.845 INFO:teuthology.orchestra.run.smithi152.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-06T03:36:54.845 INFO:teuthology.orchestra.run.smithi152.stdout:HOME_URL="https://centos.org/" 2023-12-06T03:36:54.845 INFO:teuthology.orchestra.run.smithi152.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-06T03:36:54.845 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-06T03:36:54.846 INFO:teuthology.orchestra.run.smithi152.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-06T03:36:54.846 INFO:teuthology.lock.ops:Updating smithi152.front.sepia.ceph.com on lock server 2023-12-06T03:36:54.869 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-06T03:36:54.879 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-06T03:36:54.946 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-06T03:36:54.947 DEBUG:teuthology.orchestra.run.smithi132:> test '!' -e /home/ubuntu/cephtest 2023-12-06T03:36:54.950 DEBUG:teuthology.orchestra.run.smithi152:> test '!' -e /home/ubuntu/cephtest 2023-12-06T03:36:54.970 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-06T03:36:54.976 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-06T03:36:54.976 DEBUG:teuthology.orchestra.run.smithi132:> test -z $(ls -A /var/lib/ceph) 2023-12-06T03:36:55.011 DEBUG:teuthology.orchestra.run.smithi152:> test -z $(ls -A /var/lib/ceph) 2023-12-06T03:36:55.040 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-06T03:36:55.094 INFO:teuthology.run_tasks:Running task kernel... 2023-12-06T03:36:55.110 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-06T03:36:55.111 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-06T03:36:55.111 DEBUG:teuthology.orchestra.run.smithi132:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-06T03:36:55.111 DEBUG:teuthology.orchestra.run.smithi152:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-06T03:36:55.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T03:36:55.132 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2023-12-06T03:36:55.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T03:36:55.133 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2023-12-06T03:36:55.192 INFO:teuthology.orchestra.run.smithi132.stdout:4.18.0-489.el8.x86_64 2023-12-06T03:36:55.193 INFO:teuthology.task.kernel:Running kernel on smithi132: 4.18.0-489.el8.x86_64 2023-12-06T03:36:55.193 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum install -y kernel 2023-12-06T03:36:55.194 INFO:teuthology.orchestra.run.smithi152.stdout:4.18.0-489.el8.x86_64 2023-12-06T03:36:55.194 INFO:teuthology.task.kernel:Running kernel on smithi152: 4.18.0-489.el8.x86_64 2023-12-06T03:36:55.194 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum install -y kernel 2023-12-06T03:36:58.594 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - AppStream 42 MB/s | 34 MB 00:00 2023-12-06T03:36:58.630 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 8 - AppStream 41 MB/s | 34 MB 00:00 2023-12-06T03:37:06.770 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 8 - BaseOS 28 MB/s | 55 MB 00:01 2023-12-06T03:37:06.794 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - BaseOS 28 MB/s | 55 MB 00:01 2023-12-06T03:37:14.836 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 8 - Extras 66 kB/s | 18 kB 00:00 2023-12-06T03:37:14.943 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - Extras 64 kB/s | 18 kB 00:00 2023-12-06T03:37:15.387 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-12-06T03:37:15.452 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 6.9 kB 00:00 2023-12-06T03:37:15.705 INFO:teuthology.orchestra.run.smithi152.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2023-12-06T03:37:15.779 INFO:teuthology.orchestra.run.smithi132.stdout:Copr repo for python3-asyncssh owned by ceph 20 kB/s | 3.5 kB 00:00 2023-12-06T03:37:17.934 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 7.7 MB/s | 16 MB 00:02 2023-12-06T03:37:19.088 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 5.0 MB/s | 16 MB 00:03 2023-12-06T03:37:21.798 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 693 kB/s | 24 kB 00:00 2023-12-06T03:37:22.921 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 548 kB/s | 24 kB 00:00 2023-12-06T03:37:25.406 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-06T03:37:25.406 INFO:teuthology.orchestra.run.smithi152.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-06T03:37:25.715 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:37:25.715 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:37:25.715 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2023-12-06T03:37:25.715 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:37:25.715 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:37:25.715 INFO:teuthology.orchestra.run.smithi152.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-06T03:37:25.716 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2023-12-06T03:37:25.716 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-06T03:37:25.716 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-06T03:37:25.716 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:37:25.716 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:37:25.716 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:37:25.716 INFO:teuthology.orchestra.run.smithi152.stdout:Install 3 Packages 2023-12-06T03:37:25.716 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:37:25.716 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 90 M 2023-12-06T03:37:25.716 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 96 M 2023-12-06T03:37:25.716 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:37:26.249 INFO:teuthology.orchestra.run.smithi152.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 20 MB/s | 10 MB 00:00 2023-12-06T03:37:26.606 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-06T03:37:26.606 INFO:teuthology.orchestra.run.smithi132.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-06T03:37:26.900 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:37:26.901 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:37:26.901 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2023-12-06T03:37:26.901 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:37:26.901 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:37:26.902 INFO:teuthology.orchestra.run.smithi132.stdout: kernel x86_64 4.18.0-526.el8 baseos 10 M 2023-12-06T03:37:26.902 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T03:37:26.902 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-core x86_64 4.18.0-526.el8 baseos 43 M 2023-12-06T03:37:26.902 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules x86_64 4.18.0-526.el8 baseos 36 M 2023-12-06T03:37:26.902 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:37:26.902 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:37:26.902 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:37:26.902 INFO:teuthology.orchestra.run.smithi132.stdout:Install 3 Packages 2023-12-06T03:37:26.902 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:37:26.903 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 90 M 2023-12-06T03:37:26.903 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 96 M 2023-12-06T03:37:26.903 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:37:27.175 INFO:teuthology.orchestra.run.smithi152.stdout:(2/3): kernel-core-4.18.0-526.el8.x86_64.rpm 31 MB/s | 43 MB 00:01 2023-12-06T03:37:27.681 INFO:teuthology.orchestra.run.smithi132.stdout:(1/3): kernel-4.18.0-526.el8.x86_64.rpm 17 MB/s | 10 MB 00:00 2023-12-06T03:37:27.825 INFO:teuthology.orchestra.run.smithi152.stdout:(3/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 17 MB/s | 36 MB 00:02 2023-12-06T03:37:27.825 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:37:27.825 INFO:teuthology.orchestra.run.smithi152.stdout:Total 42 MB/s | 90 MB 00:02 2023-12-06T03:37:28.499 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:37:28.731 INFO:teuthology.orchestra.run.smithi132.stdout:(2/3): kernel-modules-4.18.0-526.el8.x86_64.rpm 22 MB/s | 36 MB 00:01 2023-12-06T03:37:28.930 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:37:28.930 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:37:29.173 INFO:teuthology.orchestra.run.smithi132.stdout:(3/3): kernel-core-4.18.0-526.el8.x86_64.rpm 21 MB/s | 43 MB 00:02 2023-12-06T03:37:29.173 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:37:29.173 INFO:teuthology.orchestra.run.smithi132.stdout:Total 39 MB/s | 90 MB 00:02 2023-12-06T03:37:29.850 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:37:30.169 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:37:30.192 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:37:30.277 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:37:30.277 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:37:31.525 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:37:31.547 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:37:32.984 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:37:33.489 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-06T03:37:34.469 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:37:35.090 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-06T03:37:37.031 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-06T03:37:37.647 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-06T03:37:38.685 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 1/3 2023-12-06T03:37:39.098 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-06T03:37:41.081 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-06T03:37:41.354 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-06T03:37:42.525 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 2/3 2023-12-06T03:37:42.821 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : kernel-4.18.0-526.el8.x86_64 3/3 2023-12-06T03:38:03.511 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-06T03:38:03.535 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-06T03:38:03.889 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-06T03:38:03.889 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-06T03:38:03.889 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-06T03:38:04.748 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-core-4.18.0-526.el8.x86_64 3/3 2023-12-06T03:38:04.773 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-06T03:38:04.878 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-06T03:38:04.878 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:38:04.878 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:38:04.878 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-06T03:38:04.878 INFO:teuthology.orchestra.run.smithi152.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-06T03:38:04.878 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:38:04.878 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:38:05.023 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T03:38:05.119 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: kernel-4.18.0-526.el8.x86_64 3/3 2023-12-06T03:38:05.119 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-4.18.0-526.el8.x86_64 1/3 2023-12-06T03:38:05.119 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-core-4.18.0-526.el8.x86_64 2/3 2023-12-06T03:38:05.240 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-06T03:38:05.240 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-06T03:38:05.240 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-06T03:38:05.240 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-06T03:38:05.240 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-06T03:38:05.241 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-12-06T03:38:05.241 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi152.front.sepia.ceph.com, path=None, version=distro) 2023-12-06T03:38:05.241 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T03:38:05.450 INFO:teuthology.orchestra.run.smithi152.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-06T03:38:05.450 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-06T03:38:05.451 DEBUG:teuthology.orchestra.run.smithi152:> sudo rpm -qi grub2-tools 2023-12-06T03:38:05.484 INFO:teuthology.orchestra.run.smithi152.stdout:Name : grub2-tools 2023-12-06T03:38:05.485 INFO:teuthology.orchestra.run.smithi152.stdout:Epoch : 1 2023-12-06T03:38:05.485 INFO:teuthology.orchestra.run.smithi152.stdout:Version : 2.02 2023-12-06T03:38:05.485 INFO:teuthology.orchestra.run.smithi152.stdout:Release : 129.el8 2023-12-06T03:38:05.485 INFO:teuthology.orchestra.run.smithi152.stdout:Architecture: x86_64 2023-12-06T03:38:05.485 INFO:teuthology.orchestra.run.smithi152.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-06T03:38:05.485 INFO:teuthology.orchestra.run.smithi152.stdout:Group : System Environment/Base 2023-12-06T03:38:05.485 INFO:teuthology.orchestra.run.smithi152.stdout:Size : 9265586 2023-12-06T03:38:05.485 INFO:teuthology.orchestra.run.smithi152.stdout:License : GPLv3+ 2023-12-06T03:38:05.485 INFO:teuthology.orchestra.run.smithi152.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-06T03:38:05.485 INFO:teuthology.orchestra.run.smithi152.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-06T03:38:05.485 INFO:teuthology.orchestra.run.smithi152.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-06T03:38:05.485 INFO:teuthology.orchestra.run.smithi152.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-06T03:38:05.486 INFO:teuthology.orchestra.run.smithi152.stdout:Relocations : (not relocatable) 2023-12-06T03:38:05.486 INFO:teuthology.orchestra.run.smithi152.stdout:Packager : CentOS BuildSystem 2023-12-06T03:38:05.486 INFO:teuthology.orchestra.run.smithi152.stdout:Vendor : CentOS 2023-12-06T03:38:05.486 INFO:teuthology.orchestra.run.smithi152.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-06T03:38:05.486 INFO:teuthology.orchestra.run.smithi152.stdout:Summary : Support tools for GRUB. 2023-12-06T03:38:05.486 INFO:teuthology.orchestra.run.smithi152.stdout:Description : 2023-12-06T03:38:05.486 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:38:05.486 INFO:teuthology.orchestra.run.smithi152.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-06T03:38:05.486 INFO:teuthology.orchestra.run.smithi152.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-06T03:38:05.486 INFO:teuthology.orchestra.run.smithi152.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-06T03:38:05.486 INFO:teuthology.orchestra.run.smithi152.stdout:hardware devices. 2023-12-06T03:38:05.486 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:38:05.486 INFO:teuthology.orchestra.run.smithi152.stdout:This subpackage provides tools for support of all platforms. 2023-12-06T03:38:05.487 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-06T03:38:05.487 INFO:teuthology.task.kernel:Updating grub on smithi152 to boot 4.18.0-526.el8.x86_64 2023-12-06T03:38:05.487 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-06T03:38:05.897 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : kernel-modules-4.18.0-526.el8.x86_64 3/3 2023-12-06T03:38:05.897 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:38:05.898 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:38:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-4.18.0-526.el8.x86_64 kernel-core-4.18.0-526.el8.x86_64 2023-12-06T03:38:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: kernel-modules-4.18.0-526.el8.x86_64 2023-12-06T03:38:05.898 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:38:05.898 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:38:06.058 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T03:38:06.270 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-06T03:38:06.270 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-06T03:38:06.270 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-526.el8.x86_64 2023-12-06T03:38:06.270 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-06T03:38:06.270 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-06T03:38:06.271 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-12-06T03:38:06.271 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi132.front.sepia.ceph.com, path=None, version=distro) 2023-12-06T03:38:06.271 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q kernel | sort -rV | head -n 1 2023-12-06T03:38:06.482 INFO:teuthology.orchestra.run.smithi132.stdout:kernel-4.18.0-526.el8.x86_64 2023-12-06T03:38:06.483 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-526.el8.x86_64 2023-12-06T03:38:06.483 DEBUG:teuthology.orchestra.run.smithi132:> sudo rpm -qi grub2-tools 2023-12-06T03:38:06.516 INFO:teuthology.orchestra.run.smithi132.stdout:Name : grub2-tools 2023-12-06T03:38:06.516 INFO:teuthology.orchestra.run.smithi132.stdout:Epoch : 1 2023-12-06T03:38:06.516 INFO:teuthology.orchestra.run.smithi132.stdout:Version : 2.02 2023-12-06T03:38:06.516 INFO:teuthology.orchestra.run.smithi132.stdout:Release : 129.el8 2023-12-06T03:38:06.516 INFO:teuthology.orchestra.run.smithi132.stdout:Architecture: x86_64 2023-12-06T03:38:06.516 INFO:teuthology.orchestra.run.smithi132.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-06T03:38:06.517 INFO:teuthology.orchestra.run.smithi132.stdout:Group : System Environment/Base 2023-12-06T03:38:06.517 INFO:teuthology.orchestra.run.smithi132.stdout:Size : 9265586 2023-12-06T03:38:06.517 INFO:teuthology.orchestra.run.smithi132.stdout:License : GPLv3+ 2023-12-06T03:38:06.517 INFO:teuthology.orchestra.run.smithi132.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-06T03:38:06.517 INFO:teuthology.orchestra.run.smithi132.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-06T03:38:06.517 INFO:teuthology.orchestra.run.smithi132.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-06T03:38:06.517 INFO:teuthology.orchestra.run.smithi132.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-06T03:38:06.517 INFO:teuthology.orchestra.run.smithi132.stdout:Relocations : (not relocatable) 2023-12-06T03:38:06.517 INFO:teuthology.orchestra.run.smithi132.stdout:Packager : CentOS BuildSystem 2023-12-06T03:38:06.517 INFO:teuthology.orchestra.run.smithi132.stdout:Vendor : CentOS 2023-12-06T03:38:06.517 INFO:teuthology.orchestra.run.smithi132.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-06T03:38:06.517 INFO:teuthology.orchestra.run.smithi132.stdout:Summary : Support tools for GRUB. 2023-12-06T03:38:06.517 INFO:teuthology.orchestra.run.smithi132.stdout:Description : 2023-12-06T03:38:06.517 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:38:06.518 INFO:teuthology.orchestra.run.smithi132.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-06T03:38:06.518 INFO:teuthology.orchestra.run.smithi132.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-06T03:38:06.518 INFO:teuthology.orchestra.run.smithi132.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-06T03:38:06.518 INFO:teuthology.orchestra.run.smithi132.stdout:hardware devices. 2023-12-06T03:38:06.518 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:38:06.518 INFO:teuthology.orchestra.run.smithi132.stdout:This subpackage provides tools for support of all platforms. 2023-12-06T03:38:06.519 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-06T03:38:06.519 INFO:teuthology.task.kernel:Updating grub on smithi132 to boot 4.18.0-526.el8.x86_64 2023-12-06T03:38:06.519 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-06T03:38:06.908 INFO:teuthology.orchestra.run.smithi152.stderr:Generating grub configuration file ... 2023-12-06T03:38:08.100 INFO:teuthology.orchestra.run.smithi132.stderr:Generating grub configuration file ... 2023-12-06T03:38:09.267 INFO:teuthology.orchestra.run.smithi152.stderr:done 2023-12-06T03:38:09.268 DEBUG:teuthology.orchestra.run.smithi152:> mktemp 2023-12-06T03:38:09.291 INFO:teuthology.orchestra.run.smithi152.stdout:/tmp/tmp.1WGPywek3D 2023-12-06T03:38:09.292 DEBUG:teuthology.orchestra.run.smithi152:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.1WGPywek3D 2023-12-06T03:38:09.365 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 0666 /tmp/tmp.1WGPywek3D 2023-12-06T03:38:09.553 DEBUG:teuthology.orchestra.remote:smithi152:/tmp/tmp.1WGPywek3D is 6KB 2023-12-06T03:38:09.589 DEBUG:teuthology.orchestra.run.smithi152:> rm -fr /tmp/tmp.1WGPywek3D 2023-12-06T03:38:09.607 DEBUG:teuthology.orchestra.run.smithi152:> sudo /bin/ls /boot/loader/entries || true 2023-12-06T03:38:09.679 INFO:teuthology.orchestra.run.smithi152.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-06T03:38:09.680 INFO:teuthology.orchestra.run.smithi152.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-06T03:38:09.680 INFO:teuthology.orchestra.run.smithi152.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-06T03:38:09.680 INFO:teuthology.orchestra.run.smithi152.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-06T03:38:09.681 DEBUG:teuthology.orchestra.run.smithi152:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-06T03:38:09.831 DEBUG:teuthology.orchestra.run.smithi152:> sudo shutdown -r now 2023-12-06T03:38:10.338 INFO:teuthology.orchestra.run.smithi132.stderr:done 2023-12-06T03:38:10.339 DEBUG:teuthology.orchestra.run.smithi132:> mktemp 2023-12-06T03:38:10.362 INFO:teuthology.orchestra.run.smithi132.stdout:/tmp/tmp.VFLegOr2H0 2023-12-06T03:38:10.362 DEBUG:teuthology.orchestra.run.smithi132:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VFLegOr2H0 2023-12-06T03:38:10.438 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 0666 /tmp/tmp.VFLegOr2H0 2023-12-06T03:38:10.580 DEBUG:teuthology.orchestra.remote:smithi132:/tmp/tmp.VFLegOr2H0 is 6KB 2023-12-06T03:38:10.628 DEBUG:teuthology.orchestra.run.smithi132:> rm -fr /tmp/tmp.VFLegOr2H0 2023-12-06T03:38:10.647 DEBUG:teuthology.orchestra.run.smithi132:> sudo /bin/ls /boot/loader/entries || true 2023-12-06T03:38:10.717 INFO:teuthology.orchestra.run.smithi132.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-06T03:38:10.717 INFO:teuthology.orchestra.run.smithi132.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-06T03:38:10.717 INFO:teuthology.orchestra.run.smithi132.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-06T03:38:10.717 INFO:teuthology.orchestra.run.smithi132.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64.conf 2023-12-06T03:38:10.718 DEBUG:teuthology.orchestra.run.smithi132:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-526.el8.x86_64 2023-12-06T03:38:10.898 DEBUG:teuthology.orchestra.run.smithi132:> sudo shutdown -r now 2023-12-06T03:38:39.864 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-12-06T03:38:39.865 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-12-06T03:38:39.866 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T03:38:40.901 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-12-06T03:38:40.902 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2023-12-06T03:38:40.902 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T03:38:58.278 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-12-06T03:38:59.298 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.132 2023-12-06T03:39:07.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-12-06T03:39:07.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T03:39:08.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2023-12-06T03:39:08.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T03:39:10.338 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.152 2023-12-06T03:39:22.350 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-12-06T03:39:22.351 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T03:40:08.346 DEBUG:teuthology.orchestra.remote:timed out 2023-12-06T03:40:20.358 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2023-12-06T03:40:20.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T03:40:20.709 DEBUG:teuthology.orchestra.run.smithi132:> true 2023-12-06T03:40:21.009 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2023-12-06T03:40:21.009 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-526.el8.x86_64"... 2023-12-06T03:40:21.009 DEBUG:teuthology.orchestra.run.smithi132:> uname -r 2023-12-06T03:40:21.029 INFO:teuthology.orchestra.run.smithi132.stdout:4.18.0-526.el8.x86_64 2023-12-06T03:40:21.029 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-06T03:40:21.029 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-06T03:40:21.029 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-06T03:40:22.030 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-12-06T03:40:22.031 DEBUG:teuthology.orchestra.run.smithi132:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-06T03:40:22.109 INFO:teuthology.orchestra.run.smithi132.stdout:ttyS1 2023-12-06T03:40:22.125 DEBUG:teuthology.parallel:result is None 2023-12-06T03:40:22.353 DEBUG:teuthology.orchestra.remote:timed out 2023-12-06T03:40:37.354 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-12-06T03:40:37.355 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T03:40:37.771 DEBUG:teuthology.orchestra.run.smithi152:> true 2023-12-06T03:40:38.053 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-12-06T03:40:38.053 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-526.el8.x86_64"... 2023-12-06T03:40:38.053 DEBUG:teuthology.orchestra.run.smithi152:> uname -r 2023-12-06T03:40:38.072 INFO:teuthology.orchestra.run.smithi152.stdout:4.18.0-526.el8.x86_64 2023-12-06T03:40:38.073 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-526.el8.x86_64 vs 4.18.0-526.el8.x86_64 2023-12-06T03:40:38.073 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-06T03:40:38.073 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-06T03:40:39.073 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-12-06T03:40:39.074 DEBUG:teuthology.orchestra.run.smithi152:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-06T03:40:39.152 INFO:teuthology.orchestra.run.smithi152.stdout:ttyS1 2023-12-06T03:40:39.168 DEBUG:teuthology.parallel:result is None 2023-12-06T03:40:39.168 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-06T03:40:39.176 INFO:teuthology.task.internal:Creating test directory... 2023-12-06T03:40:39.176 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-06T03:40:39.178 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-06T03:40:39.199 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-06T03:40:39.206 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-06T03:40:39.250 INFO:teuthology.task.internal:Creating archive directory... 2023-12-06T03:40:39.250 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-06T03:40:39.253 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-06T03:40:39.435 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-06T03:40:39.443 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-06T03:40:39.443 DEBUG:teuthology.orchestra.run.smithi132:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-06T03:40:39.445 DEBUG:teuthology.orchestra.run.smithi152:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-06T03:40:39.477 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T03:40:39.488 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T03:40:39.488 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T03:40:39.499 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-06T03:40:39.500 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-06T03:40:39.507 INFO:teuthology.task.internal:Configuring sudo... 2023-12-06T03:40:39.507 DEBUG:teuthology.orchestra.run.smithi132:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-06T03:40:39.532 DEBUG:teuthology.orchestra.run.smithi152:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-06T03:40:39.570 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-06T03:40:39.579 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-06T03:40:39.580 DEBUG:teuthology.orchestra.run.smithi132:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-06T03:40:39.603 DEBUG:teuthology.orchestra.run.smithi152:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-06T03:40:39.630 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T03:40:39.703 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T03:40:39.788 DEBUG:teuthology.orchestra.run.smithi132:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T03:40:39.834 DEBUG:teuthology.orchestra.run.smithi132:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T03:40:39.876 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T03:40:39.876 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-06T03:40:39.943 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T03:40:39.992 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-06T03:40:40.051 DEBUG:teuthology.orchestra.run.smithi152:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T03:40:40.097 DEBUG:teuthology.orchestra.run.smithi152:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-06T03:40:40.141 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-06T03:40:40.141 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-06T03:40:40.209 DEBUG:teuthology.orchestra.run.smithi132:> sudo service rsyslog restart 2023-12-06T03:40:40.212 DEBUG:teuthology.orchestra.run.smithi152:> sudo service rsyslog restart 2023-12-06T03:40:40.271 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-06T03:40:40.306 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-06T03:40:40.618 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-06T03:40:40.626 INFO:teuthology.task.internal:Starting timer... 2023-12-06T03:40:40.626 INFO:teuthology.run_tasks:Running task pcp... 2023-12-06T03:40:40.654 INFO:teuthology.run_tasks:Running task selinux... 2023-12-06T03:40:40.707 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-06T03:40:40.707 DEBUG:teuthology.orchestra.run.smithi132:> sudo service auditd rotate 2023-12-06T03:40:40.782 INFO:teuthology.orchestra.run.smithi132.stdout:Rotating logs: 2023-12-06T03:40:40.783 DEBUG:teuthology.orchestra.run.smithi152:> sudo service auditd rotate 2023-12-06T03:40:40.858 INFO:teuthology.orchestra.run.smithi152.stdout:Rotating logs: 2023-12-06T03:40:40.860 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-06T03:40:40.860 DEBUG:teuthology.orchestra.run.smithi132:> /usr/sbin/getenforce 2023-12-06T03:40:40.883 INFO:teuthology.orchestra.run.smithi132.stdout:Permissive 2023-12-06T03:40:40.883 DEBUG:teuthology.orchestra.run.smithi152:> /usr/sbin/getenforce 2023-12-06T03:40:40.916 INFO:teuthology.orchestra.run.smithi152.stdout:Permissive 2023-12-06T03:40:40.916 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi132.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi152.front.sepia.ceph.com': 'permissive'} 2023-12-06T03:40:40.916 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-06T03:40:40.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T03:40:40.950 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-06T03:40:40.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T03:40:40.983 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-06T03:40:40.983 DEBUG:teuthology.orchestra.run.smithi132:> sudo /usr/sbin/setenforce permissive 2023-12-06T03:40:41.017 DEBUG:teuthology.orchestra.run.smithi152:> sudo /usr/sbin/setenforce permissive 2023-12-06T03:40:41.050 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-06T03:40:41.059 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main was just updated or references a specific commit; assuming it is current 2023-12-06T03:40:41.060 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-06T03:40:41.078 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-06T03:40:41.079 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi132.front.sepia.ceph.com,smithi152.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-06T03:48:04.612 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi132.front.sepia.ceph.com'), Remote(name='ubuntu@smithi152.front.sepia.ceph.com')] 2023-12-06T03:48:04.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi132.front.sepia.ceph.com' 2023-12-06T03:48:04.613 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi132.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T03:48:04.683 DEBUG:teuthology.orchestra.run.smithi132:> true 2023-12-06T03:48:04.752 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi132.front.sepia.ceph.com' 2023-12-06T03:48:04.753 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-12-06T03:48:04.754 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi152.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-06T03:48:04.826 DEBUG:teuthology.orchestra.run.smithi152:> true 2023-12-06T03:48:04.894 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi152.front.sepia.ceph.com' 2023-12-06T03:48:04.894 INFO:teuthology.run_tasks:Running task clock... 2023-12-06T03:48:04.906 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-06T03:48:04.907 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-06T03:48:04.907 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-06T03:48:04.910 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-06T03:48:04.910 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-06T03:48:04.940 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-06T03:48:04.954 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-06T03:48:04.968 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-06T03:48:04.982 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-06T03:48:05.021 INFO:teuthology.orchestra.run.smithi132.stderr:sudo: ntpd: command not found 2023-12-06T03:48:05.033 INFO:teuthology.orchestra.run.smithi132.stdout:506 Cannot talk to daemon 2023-12-06T03:48:05.046 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-06T03:48:05.049 INFO:teuthology.orchestra.run.smithi152.stderr:sudo: ntpd: command not found 2023-12-06T03:48:05.059 INFO:teuthology.orchestra.run.smithi132.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-06T03:48:05.060 INFO:teuthology.orchestra.run.smithi152.stdout:506 Cannot talk to daemon 2023-12-06T03:48:05.074 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-06T03:48:05.087 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-06T03:48:05.131 INFO:teuthology.orchestra.run.smithi132.stderr:bash: ntpq: command not found 2023-12-06T03:48:05.134 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-06T03:48:05.134 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2023-12-06T03:48:05.134 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T03:48:05.134 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T03:48:05.134 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T03:48:05.135 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T03:48:05.153 INFO:teuthology.orchestra.run.smithi152.stderr:bash: ntpq: command not found 2023-12-06T03:48:05.157 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-06T03:48:05.157 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2023-12-06T03:48:05.157 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T03:48:05.157 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T03:48:05.157 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T03:48:05.157 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T03:48:05.158 INFO:teuthology.run_tasks:Running task pexec... 2023-12-06T03:48:05.167 INFO:teuthology.task.pexec:Executing custom commands... 2023-12-06T03:48:05.167 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T03:48:05.168 DEBUG:teuthology.orchestra.run.smithi132:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-06T03:48:05.168 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi152.front.sepia.ceph.com 2023-12-06T03:48:05.168 DEBUG:teuthology.orchestra.run.smithi152:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-06T03:48:06.057 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:35 ago on Wed 06 Dec 2023 03:46:30 AM UTC. 2023-12-06T03:48:06.075 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:34 ago on Wed 06 Dec 2023 03:46:31 AM UTC. 2023-12-06T03:48:08.158 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:48:08.159 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:48:08.159 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2023-12-06T03:48:08.159 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:48:08.159 INFO:teuthology.orchestra.run.smithi132.stdout:Resetting modules: 2023-12-06T03:48:08.159 INFO:teuthology.orchestra.run.smithi132.stdout: container-tools 2023-12-06T03:48:08.159 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:48:08.160 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:48:08.160 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:48:08.160 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:48:08.176 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:48:08.176 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:48:08.176 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2023-12-06T03:48:08.177 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:48:08.177 INFO:teuthology.orchestra.run.smithi152.stdout:Resetting modules: 2023-12-06T03:48:08.177 INFO:teuthology.orchestra.run.smithi152.stdout: container-tools 2023-12-06T03:48:08.177 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:48:08.177 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:48:08.177 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:48:08.177 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:48:08.418 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:48:08.426 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:48:09.345 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:38 ago on Wed 06 Dec 2023 03:46:31 AM UTC. 2023-12-06T03:48:09.360 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:39 ago on Wed 06 Dec 2023 03:46:30 AM UTC. 2023-12-06T03:48:11.454 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:48:11.475 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:48:11.638 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:48:11.638 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T03:48:11.638 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:48:11.638 INFO:teuthology.orchestra.run.smithi132.stdout:Installing group/module packages: 2023-12-06T03:48:11.638 INFO:teuthology.orchestra.run.smithi132.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout:Installing weak dependencies: 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout:Installing module profiles: 2023-12-06T03:48:11.639 INFO:teuthology.orchestra.run.smithi132.stdout: container-tools/common 2023-12-06T03:48:11.640 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:48:11.640 INFO:teuthology.orchestra.run.smithi132.stdout:Enabling module streams: 2023-12-06T03:48:11.640 INFO:teuthology.orchestra.run.smithi132.stdout: container-tools rhel8 2023-12-06T03:48:11.640 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:48:11.640 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:48:11.640 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:48:11.640 INFO:teuthology.orchestra.run.smithi132.stdout:Install 13 Packages 2023-12-06T03:48:11.640 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:48:11.641 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 27 M 2023-12-06T03:48:11.641 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 89 M 2023-12-06T03:48:11.641 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:48:11.659 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:48:11.659 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2023-12-06T03:48:11.659 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:48:11.659 INFO:teuthology.orchestra.run.smithi152.stdout:Installing group/module packages: 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout:Installing weak dependencies: 2023-12-06T03:48:11.660 INFO:teuthology.orchestra.run.smithi152.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-12-06T03:48:11.661 INFO:teuthology.orchestra.run.smithi152.stdout:Installing module profiles: 2023-12-06T03:48:11.661 INFO:teuthology.orchestra.run.smithi152.stdout: container-tools/common 2023-12-06T03:48:11.661 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:48:11.661 INFO:teuthology.orchestra.run.smithi152.stdout:Enabling module streams: 2023-12-06T03:48:11.661 INFO:teuthology.orchestra.run.smithi152.stdout: container-tools rhel8 2023-12-06T03:48:11.661 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:48:11.661 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:48:11.661 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:48:11.661 INFO:teuthology.orchestra.run.smithi152.stdout:Install 13 Packages 2023-12-06T03:48:11.661 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:48:11.662 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 27 M 2023-12-06T03:48:11.662 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 89 M 2023-12-06T03:48:11.662 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:48:11.939 INFO:teuthology.orchestra.run.smithi132.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 6.5 MB/s | 750 kB 00:00 2023-12-06T03:48:11.969 INFO:teuthology.orchestra.run.smithi152.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 6.1 MB/s | 750 kB 00:00 2023-12-06T03:48:11.972 INFO:teuthology.orchestra.run.smithi132.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 6.8 MB/s | 1.0 MB 00:00 2023-12-06T03:48:11.998 INFO:teuthology.orchestra.run.smithi132.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 646 kB/s | 38 kB 00:00 2023-12-06T03:48:12.004 INFO:teuthology.orchestra.run.smithi152.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 6.5 MB/s | 1.0 MB 00:00 2023-12-06T03:48:12.023 INFO:teuthology.orchestra.run.smithi132.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 4.7 MB/s | 241 kB 00:00 2023-12-06T03:48:12.028 INFO:teuthology.orchestra.run.smithi152.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 648 kB/s | 38 kB 00:00 2023-12-06T03:48:12.054 INFO:teuthology.orchestra.run.smithi152.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 4.7 MB/s | 241 kB 00:00 2023-12-06T03:48:12.057 INFO:teuthology.orchestra.run.smithi132.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 5.2 MB/s | 178 kB 00:00 2023-12-06T03:48:12.087 INFO:teuthology.orchestra.run.smithi152.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 5.2 MB/s | 178 kB 00:00 2023-12-06T03:48:12.091 INFO:teuthology.orchestra.run.smithi132.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 743 kB/s | 25 kB 00:00 2023-12-06T03:48:12.112 INFO:teuthology.orchestra.run.smithi152.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 1.0 MB/s | 25 kB 00:00 2023-12-06T03:48:12.157 INFO:teuthology.orchestra.run.smithi132.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 25 MB/s | 3.9 MB 00:00 2023-12-06T03:48:12.171 INFO:teuthology.orchestra.run.smithi152.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 27 MB/s | 3.9 MB 00:00 2023-12-06T03:48:12.183 INFO:teuthology.orchestra.run.smithi132.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.0 MB/s | 94 kB 00:00 2023-12-06T03:48:12.196 INFO:teuthology.orchestra.run.smithi152.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.1 MB/s | 94 kB 00:00 2023-12-06T03:48:12.305 INFO:teuthology.orchestra.run.smithi152.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 21 MB/s | 9.6 MB 00:00 2023-12-06T03:48:12.317 INFO:teuthology.orchestra.run.smithi132.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 20 MB/s | 9.6 MB 00:00 2023-12-06T03:48:12.330 INFO:teuthology.orchestra.run.smithi152.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 2.1 MB/s | 53 kB 00:00 2023-12-06T03:48:12.367 INFO:teuthology.orchestra.run.smithi132.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 14 MB/s | 2.5 MB 00:00 2023-12-06T03:48:12.372 INFO:teuthology.orchestra.run.smithi152.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 14 MB/s | 2.5 MB 00:00 2023-12-06T03:48:12.389 INFO:teuthology.orchestra.run.smithi152.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 700 kB/s | 41 kB 00:00 2023-12-06T03:48:12.392 INFO:teuthology.orchestra.run.smithi132.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 698 kB/s | 53 kB 00:00 2023-12-06T03:48:12.417 INFO:teuthology.orchestra.run.smithi132.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 820 kB/s | 41 kB 00:00 2023-12-06T03:48:12.498 INFO:teuthology.orchestra.run.smithi152.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 25 MB/s | 8.1 MB 00:00 2023-12-06T03:48:12.498 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:48:12.498 INFO:teuthology.orchestra.run.smithi152.stdout:Total 32 MB/s | 27 MB 00:00 2023-12-06T03:48:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 21 MB/s | 8.1 MB 00:00 2023-12-06T03:48:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:48:12.543 INFO:teuthology.orchestra.run.smithi132.stdout:Total 29 MB/s | 27 MB 00:00 2023-12-06T03:48:12.764 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:48:12.785 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:48:12.786 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:48:12.809 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:48:12.830 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:48:12.830 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:48:13.048 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:48:13.051 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:48:13.075 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:48:13.078 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:48:13.735 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:48:13.736 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:48:13.929 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-12-06T03:48:13.946 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-12-06T03:48:14.107 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-12-06T03:48:14.116 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-12-06T03:48:14.290 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-12-06T03:48:14.332 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-12-06T03:48:14.512 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-06T03:48:14.542 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-06T03:48:15.098 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-12-06T03:48:15.110 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-12-06T03:48:16.062 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-06T03:48:16.111 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-06T03:48:16.306 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-12-06T03:48:16.396 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-12-06T03:48:16.525 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-12-06T03:48:16.610 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-12-06T03:48:16.779 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-12-06T03:48:16.937 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-12-06T03:48:17.205 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-12-06T03:48:17.337 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-12-06T03:48:18.074 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-06T03:48:18.202 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-06T03:48:18.262 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-12-06T03:48:18.421 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-12-06T03:48:18.432 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-06T03:48:18.590 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-06T03:48:18.868 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-06T03:48:18.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-12-06T03:48:18.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-12-06T03:48:18.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-12-06T03:48:18.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-06T03:48:18.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-12-06T03:48:18.868 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-06T03:48:18.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-12-06T03:48:18.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-12-06T03:48:18.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-12-06T03:48:18.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-12-06T03:48:18.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-06T03:48:18.869 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-12-06T03:48:19.042 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-06T03:48:19.042 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-12-06T03:48:19.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-12-06T03:48:19.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-12-06T03:48:19.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-06T03:48:19.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-12-06T03:48:19.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-06T03:48:19.043 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-12-06T03:48:19.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-12-06T03:48:19.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-12-06T03:48:19.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-12-06T03:48:19.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-06T03:48:19.044 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-12-06T03:48:19.648 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-12-06T03:48:19.648 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:48:19.648 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:48:19.648 INFO:teuthology.orchestra.run.smithi132.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-06T03:48:19.649 INFO:teuthology.orchestra.run.smithi132.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-06T03:48:19.649 INFO:teuthology.orchestra.run.smithi132.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-12-06T03:48:19.649 INFO:teuthology.orchestra.run.smithi132.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-12-06T03:48:19.649 INFO:teuthology.orchestra.run.smithi132.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-12-06T03:48:19.649 INFO:teuthology.orchestra.run.smithi132.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-12-06T03:48:19.649 INFO:teuthology.orchestra.run.smithi132.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-12-06T03:48:19.649 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-12-06T03:48:19.649 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-12-06T03:48:19.649 INFO:teuthology.orchestra.run.smithi132.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-12-06T03:48:19.649 INFO:teuthology.orchestra.run.smithi132.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-12-06T03:48:19.649 INFO:teuthology.orchestra.run.smithi132.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-12-06T03:48:19.649 INFO:teuthology.orchestra.run.smithi132.stdout: yajl-2.1.0-12.el8.x86_64 2023-12-06T03:48:19.649 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:48:19.649 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:48:19.880 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-12-06T03:48:19.880 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:48:19.880 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:48:19.880 INFO:teuthology.orchestra.run.smithi152.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-06T03:48:19.880 INFO:teuthology.orchestra.run.smithi152.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-06T03:48:19.880 INFO:teuthology.orchestra.run.smithi152.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-12-06T03:48:19.880 INFO:teuthology.orchestra.run.smithi152.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-12-06T03:48:19.880 INFO:teuthology.orchestra.run.smithi152.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-12-06T03:48:19.880 INFO:teuthology.orchestra.run.smithi152.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-12-06T03:48:19.881 INFO:teuthology.orchestra.run.smithi152.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-12-06T03:48:19.881 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-12-06T03:48:19.881 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-12-06T03:48:19.881 INFO:teuthology.orchestra.run.smithi152.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-12-06T03:48:19.881 INFO:teuthology.orchestra.run.smithi152.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-12-06T03:48:19.881 INFO:teuthology.orchestra.run.smithi152.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-12-06T03:48:19.881 INFO:teuthology.orchestra.run.smithi152.stdout: yajl-2.1.0-12.el8.x86_64 2023-12-06T03:48:19.881 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:48:19.881 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:48:19.925 DEBUG:teuthology.parallel:result is None 2023-12-06T03:48:20.128 DEBUG:teuthology.parallel:result is None 2023-12-06T03:48:20.128 INFO:teuthology.run_tasks:Running task pexec... 2023-12-06T03:48:20.136 INFO:teuthology.task.pexec:Executing custom commands... 2023-12-06T03:48:20.136 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T03:48:20.137 DEBUG:teuthology.orchestra.run.smithi132:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-06T03:48:20.137 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi152.front.sepia.ceph.com 2023-12-06T03:48:20.137 DEBUG:teuthology.orchestra.run.smithi152:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-06T03:48:20.282 DEBUG:teuthology.parallel:result is None 2023-12-06T03:48:20.343 DEBUG:teuthology.parallel:result is None 2023-12-06T03:48:20.344 INFO:teuthology.run_tasks:Running task install... 2023-12-06T03:48:20.352 DEBUG:teuthology.task.install:project ceph 2023-12-06T03:48:20.352 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'}} 2023-12-06T03:48:20.352 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'} 2023-12-06T03:48:20.352 INFO:teuthology.task.install:Using flavor: default 2023-12-06T03:48:20.358 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-06T03:48:20.358 INFO:teuthology.task.install:extra packages: [] 2023-12-06T03:48:20.359 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-12-06T03:48:20.359 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-06T03:48:20.359 INFO:teuthology.packaging:ref: None 2023-12-06T03:48:20.359 INFO:teuthology.packaging:tag: v16.2.4 2023-12-06T03:48:20.359 INFO:teuthology.packaging:branch: None 2023-12-06T03:48:20.359 INFO:teuthology.packaging:sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b 2023-12-06T03:48:20.561 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-06T03:48:20.562 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-06T03:48:20.564 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-12-06T03:48:20.564 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-06T03:48:20.564 INFO:teuthology.packaging:ref: None 2023-12-06T03:48:20.564 INFO:teuthology.packaging:tag: v16.2.4 2023-12-06T03:48:20.565 INFO:teuthology.packaging:branch: None 2023-12-06T03:48:20.565 INFO:teuthology.packaging:sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b 2023-12-06T03:48:20.750 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-06T03:48:20.750 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-06T03:48:20.903 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-12-06T03:48:20.903 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-12-06T03:48:20.945 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-06T03:48:20.946 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T03:48:20.946 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-06T03:48:20.976 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-06T03:48:20.976 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-06T03:48:20.976 INFO:teuthology.packaging:ref: None 2023-12-06T03:48:20.976 INFO:teuthology.packaging:tag: v16.2.4 2023-12-06T03:48:20.976 INFO:teuthology.packaging:branch: None 2023-12-06T03:48:20.976 INFO:teuthology.packaging:sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b 2023-12-06T03:48:20.976 DEBUG:teuthology.orchestra.run.smithi132:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-06T03:48:20.982 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-12-06T03:48:20.982 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-12-06T03:48:21.006 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-06T03:48:21.007 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-06T03:48:21.007 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-06T03:48:21.036 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-06T03:48:21.036 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-06T03:48:21.037 INFO:teuthology.packaging:ref: None 2023-12-06T03:48:21.037 INFO:teuthology.packaging:tag: v16.2.4 2023-12-06T03:48:21.037 INFO:teuthology.packaging:branch: None 2023-12-06T03:48:21.037 INFO:teuthology.packaging:sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b 2023-12-06T03:48:21.037 DEBUG:teuthology.orchestra.run.smithi152:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-06T03:48:21.045 DEBUG:teuthology.orchestra.run.smithi132:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-06T03:48:21.105 DEBUG:teuthology.orchestra.run.smithi152:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-06T03:48:21.122 DEBUG:teuthology.orchestra.run.smithi132:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-06T03:48:21.183 DEBUG:teuthology.orchestra.run.smithi152:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-06T03:48:21.188 INFO:teuthology.orchestra.run.smithi132.stdout:check_obsoletes = 1 2023-12-06T03:48:21.189 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum clean all 2023-12-06T03:48:21.250 INFO:teuthology.orchestra.run.smithi152.stdout:check_obsoletes = 1 2023-12-06T03:48:21.251 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum clean all 2023-12-06T03:48:21.452 INFO:teuthology.orchestra.run.smithi132.stdout:76 files removed 2023-12-06T03:48:21.481 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-radosgw 2023-12-06T03:48:21.520 INFO:teuthology.orchestra.run.smithi152.stdout:76 files removed 2023-12-06T03:48:21.548 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-radosgw 2023-12-06T03:48:21.951 INFO:teuthology.orchestra.run.smithi132.stdout:ceph packages for x86_64 1.1 MB/s | 74 kB 00:00 2023-12-06T03:48:22.060 INFO:teuthology.orchestra.run.smithi132.stdout:ceph noarch packages 212 kB/s | 15 kB 00:00 2023-12-06T03:48:22.337 INFO:teuthology.orchestra.run.smithi132.stdout:ceph source packages 7.0 kB/s | 1.7 kB 00:00 2023-12-06T03:48:22.439 INFO:teuthology.orchestra.run.smithi152.stdout:ceph packages for x86_64 153 kB/s | 74 kB 00:00 2023-12-06T03:48:22.556 INFO:teuthology.orchestra.run.smithi152.stdout:ceph noarch packages 193 kB/s | 15 kB 00:00 2023-12-06T03:48:22.657 INFO:teuthology.orchestra.run.smithi152.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2023-12-06T03:48:23.437 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - AppStream 32 MB/s | 34 MB 00:01 2023-12-06T03:48:23.490 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 8 - AppStream 42 MB/s | 34 MB 00:00 2023-12-06T03:48:30.996 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - BaseOS 41 MB/s | 55 MB 00:01 2023-12-06T03:48:31.724 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 8 - BaseOS 29 MB/s | 55 MB 00:01 2023-12-06T03:48:39.271 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - Extras 74 kB/s | 18 kB 00:00 2023-12-06T03:48:39.853 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-12-06T03:48:40.066 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 8 - Extras 46 kB/s | 18 kB 00:00 2023-12-06T03:48:40.466 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 8 - Extras common packages 31 kB/s | 6.9 kB 00:00 2023-12-06T03:48:40.714 INFO:teuthology.orchestra.run.smithi132.stdout:CentOS Stream 8 - PowerTools 9.3 MB/s | 6.6 MB 00:00 2023-12-06T03:48:41.301 INFO:teuthology.orchestra.run.smithi152.stdout:CentOS Stream 8 - PowerTools 9.7 MB/s | 6.6 MB 00:00 2023-12-06T03:48:42.207 INFO:teuthology.orchestra.run.smithi132.stdout:Copr repo for python3-asyncssh owned by ceph 19 kB/s | 3.5 kB 00:00 2023-12-06T03:48:42.752 INFO:teuthology.orchestra.run.smithi152.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-12-06T03:48:43.694 INFO:teuthology.orchestra.run.smithi152.stdout:Extra Packages for Enterprise Linux 20 MB/s | 16 MB 00:00 2023-12-06T03:48:44.983 INFO:teuthology.orchestra.run.smithi132.stdout:Extra Packages for Enterprise Linux 6.1 MB/s | 16 MB 00:02 2023-12-06T03:48:47.552 INFO:teuthology.orchestra.run.smithi152.stdout:lab-extras 547 kB/s | 24 kB 00:00 2023-12-06T03:48:49.126 INFO:teuthology.orchestra.run.smithi132.stdout:lab-extras 190 kB/s | 24 kB 00:00 2023-12-06T03:48:52.135 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:48:52.138 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:48:52.139 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2023-12-06T03:48:52.139 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:48:52.139 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:48:52.139 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-12-06T03:48:52.139 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrading: 2023-12-06T03:48:52.139 INFO:teuthology.orchestra.run.smithi152.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-06T03:48:52.139 INFO:teuthology.orchestra.run.smithi152.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-12-06T03:48:52.139 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2023-12-06T03:48:52.139 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-12-06T03:48:52.139 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-12-06T03:48:52.140 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-12-06T03:48:52.140 INFO:teuthology.orchestra.run.smithi152.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-12-06T03:48:52.140 INFO:teuthology.orchestra.run.smithi152.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-12-06T03:48:52.140 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-06T03:48:52.140 INFO:teuthology.orchestra.run.smithi152.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-06T03:48:52.140 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-12-06T03:48:52.140 INFO:teuthology.orchestra.run.smithi152.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-06T03:48:52.140 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-12-06T03:48:52.140 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-06T03:48:52.140 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-12-06T03:48:52.140 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-12-06T03:48:52.141 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-12-06T03:48:52.141 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-06T03:48:52.141 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-06T03:48:52.141 INFO:teuthology.orchestra.run.smithi152.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-12-06T03:48:52.141 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-12-06T03:48:52.141 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-12-06T03:48:52.141 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-12-06T03:48:52.141 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-06T03:48:52.141 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-12-06T03:48:52.141 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-12-06T03:48:52.141 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-12-06T03:48:52.141 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:48:52.141 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:48:52.142 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:48:52.142 INFO:teuthology.orchestra.run.smithi152.stdout:Install 25 Packages 2023-12-06T03:48:52.142 INFO:teuthology.orchestra.run.smithi152.stdout:Upgrade 2 Packages 2023-12-06T03:48:52.142 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:48:52.142 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 59 M 2023-12-06T03:48:52.142 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:48:52.638 INFO:teuthology.orchestra.run.smithi152.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 44 MB/s | 6.0 MB 00:00 2023-12-06T03:48:52.671 INFO:teuthology.orchestra.run.smithi152.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 740 kB/s | 25 kB 00:00 2023-12-06T03:48:52.713 INFO:teuthology.orchestra.run.smithi152.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2023-12-06T03:48:52.747 INFO:teuthology.orchestra.run.smithi152.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-12-06T03:48:52.839 INFO:teuthology.orchestra.run.smithi152.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 44 MB/s | 4.1 MB 00:00 2023-12-06T03:48:52.864 INFO:teuthology.orchestra.run.smithi152.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-12-06T03:48:52.890 INFO:teuthology.orchestra.run.smithi152.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-12-06T03:48:52.906 INFO:teuthology.orchestra.run.smithi152.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 13 MB/s | 213 kB 00:00 2023-12-06T03:48:52.940 INFO:teuthology.orchestra.run.smithi152.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-12-06T03:48:52.974 INFO:teuthology.orchestra.run.smithi152.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-12-06T03:48:52.999 INFO:teuthology.orchestra.run.smithi152.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-12-06T03:48:53.166 INFO:teuthology.orchestra.run.smithi152.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 19 MB/s | 13 MB 00:00 2023-12-06T03:48:53.191 INFO:teuthology.orchestra.run.smithi152.stdout:(13/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 757 kB/s | 145 kB 00:00 2023-12-06T03:48:53.241 INFO:teuthology.orchestra.run.smithi152.stdout:(14/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 5.1 MB/s | 259 kB 00:00 2023-12-06T03:48:53.267 INFO:teuthology.orchestra.run.smithi152.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 1.7 MB/s | 44 kB 00:00 2023-12-06T03:48:53.292 INFO:teuthology.orchestra.run.smithi152.stdout:(16/27): librdkafka-0.11.4-3.el8.x86_64.rpm 2.8 MB/s | 354 kB 00:00 2023-12-06T03:48:53.534 INFO:teuthology.orchestra.run.smithi152.stdout:(17/27): ceph-common-16.2.4-0.el8.x86_64.rpm 23 MB/s | 24 MB 00:01 2023-12-06T03:48:53.576 INFO:teuthology.orchestra.run.smithi152.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 137 kB/s | 42 kB 00:00 2023-12-06T03:48:53.593 INFO:teuthology.orchestra.run.smithi152.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 156 kB/s | 47 kB 00:00 2023-12-06T03:48:53.668 INFO:teuthology.orchestra.run.smithi152.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 600 kB/s | 80 kB 00:00 2023-12-06T03:48:53.745 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:48:53.748 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:48:53.748 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T03:48:53.748 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:48:53.748 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:48:53.748 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-12-06T03:48:53.748 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrading: 2023-12-06T03:48:53.748 INFO:teuthology.orchestra.run.smithi132.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-06T03:48:53.748 INFO:teuthology.orchestra.run.smithi132.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-12-06T03:48:53.748 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T03:48:53.749 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-12-06T03:48:53.749 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-12-06T03:48:53.749 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-12-06T03:48:53.749 INFO:teuthology.orchestra.run.smithi132.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-12-06T03:48:53.749 INFO:teuthology.orchestra.run.smithi132.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-12-06T03:48:53.749 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-06T03:48:53.749 INFO:teuthology.orchestra.run.smithi132.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-06T03:48:53.749 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-12-06T03:48:53.749 INFO:teuthology.orchestra.run.smithi132.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-06T03:48:53.749 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-12-06T03:48:53.749 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-06T03:48:53.749 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-12-06T03:48:53.750 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-12-06T03:48:53.750 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-12-06T03:48:53.750 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-06T03:48:53.750 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-06T03:48:53.750 INFO:teuthology.orchestra.run.smithi132.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-12-06T03:48:53.750 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-12-06T03:48:53.750 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-12-06T03:48:53.750 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-12-06T03:48:53.750 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-06T03:48:53.750 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-12-06T03:48:53.750 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-12-06T03:48:53.750 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-12-06T03:48:53.750 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:48:53.750 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:48:53.751 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:48:53.751 INFO:teuthology.orchestra.run.smithi132.stdout:Install 25 Packages 2023-12-06T03:48:53.751 INFO:teuthology.orchestra.run.smithi132.stdout:Upgrade 2 Packages 2023-12-06T03:48:53.751 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:48:53.751 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 59 M 2023-12-06T03:48:53.751 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:48:53.794 INFO:teuthology.orchestra.run.smithi152.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 535 kB/s | 116 kB 00:00 2023-12-06T03:48:53.819 INFO:teuthology.orchestra.run.smithi152.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 1.2 MB/s | 181 kB 00:00 2023-12-06T03:48:53.836 INFO:teuthology.orchestra.run.smithi152.stdout:(23/27): liboath-2.6.2-3.el8.x86_64.rpm 1.4 MB/s | 59 kB 00:00 2023-12-06T03:48:53.861 INFO:teuthology.orchestra.run.smithi152.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.1 MB/s | 306 kB 00:00 2023-12-06T03:48:53.886 INFO:teuthology.orchestra.run.smithi152.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.1 MB/s | 75 kB 00:00 2023-12-06T03:48:53.962 INFO:teuthology.orchestra.run.smithi152.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 30 MB/s | 3.7 MB 00:00 2023-12-06T03:48:54.028 INFO:teuthology.orchestra.run.smithi152.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 24 MB/s | 4.0 MB 00:00 2023-12-06T03:48:54.030 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:48:54.030 INFO:teuthology.orchestra.run.smithi152.stdout:Total 31 MB/s | 59 MB 00:01 2023-12-06T03:48:54.095 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:48:54.143 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:48:54.143 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:48:54.310 INFO:teuthology.orchestra.run.smithi132.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 36 MB/s | 6.0 MB 00:00 2023-12-06T03:48:54.335 INFO:teuthology.orchestra.run.smithi132.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 25 kB 00:00 2023-12-06T03:48:54.377 INFO:teuthology.orchestra.run.smithi132.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2023-12-06T03:48:54.411 INFO:teuthology.orchestra.run.smithi132.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-12-06T03:48:54.479 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:48:54.483 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:48:54.562 INFO:teuthology.orchestra.run.smithi132.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 27 MB/s | 4.1 MB 00:00 2023-12-06T03:48:54.587 INFO:teuthology.orchestra.run.smithi132.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-12-06T03:48:54.612 INFO:teuthology.orchestra.run.smithi132.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-12-06T03:48:54.754 INFO:teuthology.orchestra.run.smithi132.stdout:(8/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 21 MB/s | 13 MB 00:00 2023-12-06T03:48:54.779 INFO:teuthology.orchestra.run.smithi132.stdout:(9/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 1.2 MB/s | 213 kB 00:00 2023-12-06T03:48:54.805 INFO:teuthology.orchestra.run.smithi132.stdout:(10/27): python3-rados-16.2.4-0.el8.x86_64.rpm 7.5 MB/s | 387 kB 00:00 2023-12-06T03:48:54.830 INFO:teuthology.orchestra.run.smithi132.stdout:(11/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 7.1 MB/s | 368 kB 00:00 2023-12-06T03:48:54.855 INFO:teuthology.orchestra.run.smithi132.stdout:(12/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 114 kB 00:00 2023-12-06T03:48:55.145 INFO:teuthology.orchestra.run.smithi132.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:01 2023-12-06T03:48:55.173 INFO:teuthology.orchestra.run.smithi132.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 424 kB/s | 145 kB 00:00 2023-12-06T03:48:55.206 INFO:teuthology.orchestra.run.smithi132.stdout:(15/27): librdkafka-0.11.4-3.el8.x86_64.rpm 1.0 MB/s | 354 kB 00:00 2023-12-06T03:48:55.232 INFO:teuthology.orchestra.run.smithi132.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 749 kB/s | 44 kB 00:00 2023-12-06T03:48:55.265 INFO:teuthology.orchestra.run.smithi132.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.1 MB/s | 259 kB 00:00 2023-12-06T03:48:55.316 INFO:teuthology.orchestra.run.smithi132.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 387 kB/s | 42 kB 00:00 2023-12-06T03:48:55.341 INFO:teuthology.orchestra.run.smithi132.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 431 kB/s | 47 kB 00:00 2023-12-06T03:48:55.366 INFO:teuthology.orchestra.run.smithi132.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 798 kB/s | 80 kB 00:00 2023-12-06T03:48:55.601 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:48:55.844 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-12-06T03:48:55.942 INFO:teuthology.orchestra.run.smithi132.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 186 kB/s | 116 kB 00:00 2023-12-06T03:48:55.983 INFO:teuthology.orchestra.run.smithi132.stdout:(22/27): leveldb-1.22-1.el8.x86_64.rpm 293 kB/s | 181 kB 00:00 2023-12-06T03:48:56.009 INFO:teuthology.orchestra.run.smithi132.stdout:(23/27): gperftools-libs-2.7-9.el8.x86_64.rpm 459 kB/s | 306 kB 00:00 2023-12-06T03:48:56.050 INFO:teuthology.orchestra.run.smithi132.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 541 kB/s | 59 kB 00:00 2023-12-06T03:48:56.057 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-12-06T03:48:56.126 INFO:teuthology.orchestra.run.smithi132.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2023-12-06T03:48:56.151 INFO:teuthology.orchestra.run.smithi132.stdout:(26/27): libunwind-1.3.1-3.el8.x86_64.rpm 448 kB/s | 75 kB 00:00 2023-12-06T03:48:56.210 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-06T03:48:56.239 INFO:teuthology.orchestra.run.smithi132.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 21 MB/s | 4.0 MB 00:00 2023-12-06T03:48:56.244 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:48:56.245 INFO:teuthology.orchestra.run.smithi132.stdout:Total 24 MB/s | 59 MB 00:02 2023-12-06T03:48:56.315 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:48:56.363 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:48:56.363 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:48:56.618 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-06T03:48:56.710 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:48:56.713 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:48:56.785 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-06T03:48:56.863 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-06T03:48:57.084 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-12-06T03:48:57.251 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-06T03:48:57.315 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-06T03:48:57.507 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-12-06T03:48:57.676 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-12-06T03:48:57.819 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:48:58.027 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-12-06T03:48:58.099 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-12-06T03:48:58.231 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-12-06T03:48:58.234 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-06T03:48:58.306 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-06T03:48:58.410 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-06T03:48:58.481 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-12-06T03:48:58.655 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-06T03:48:58.738 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-06T03:48:58.817 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-12-06T03:48:59.011 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-06T03:48:59.110 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-12-06T03:48:59.157 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-12-06T03:48:59.319 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-12-06T03:48:59.345 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-12-06T03:48:59.517 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-06T03:48:59.518 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-06T03:48:59.582 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-06T03:48:59.734 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-06T03:48:59.806 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-12-06T03:49:00.009 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-12-06T03:49:00.168 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-06T03:49:00.185 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-12-06T03:49:00.375 INFO:teuthology.orchestra.run.smithi152.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-06T03:49:00.480 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-06T03:49:00.608 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-12-06T03:49:00.637 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-12-06T03:49:00.743 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-06T03:49:00.821 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-12-06T03:49:00.835 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-12-06T03:49:00.967 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-06T03:49:00.990 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-12-06T03:49:01.071 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-06T03:49:01.179 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-06T03:49:01.245 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-12-06T03:49:01.251 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-12-06T03:49:01.416 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-12-06T03:49:01.417 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-12-06T03:49:01.569 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-12-06T03:49:01.591 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-12-06T03:49:01.760 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-06T03:49:01.809 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-12-06T03:49:01.839 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-12-06T03:49:01.968 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-06T03:49:02.401 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-12-06T03:49:02.584 INFO:teuthology.orchestra.run.smithi132.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-06T03:49:02.672 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-12-06T03:49:02.888 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-12-06T03:49:03.103 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-12-06T03:49:03.243 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-06T03:49:03.321 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-12-06T03:49:03.493 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-12-06T03:49:03.667 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-12-06T03:49:03.883 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-12-06T03:49:04.076 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-12-06T03:49:04.278 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-06T03:49:04.501 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-06T03:49:05.213 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-06T03:49:05.493 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-06T03:49:05.594 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-06T03:49:05.766 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-06T03:49:06.647 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-06T03:49:06.859 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-06T03:49:07.589 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-12-06T03:49:07.884 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-06T03:49:07.972 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-12-06T03:49:08.109 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-06T03:49:24.228 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-06T03:49:24.228 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /sys 2023-12-06T03:49:24.228 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /proc 2023-12-06T03:49:24.229 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /dev 2023-12-06T03:49:24.229 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /run 2023-12-06T03:49:24.229 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /mnt 2023-12-06T03:49:24.229 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /var/tmp 2023-12-06T03:49:24.229 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /home 2023-12-06T03:49:24.229 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /tmp 2023-12-06T03:49:24.229 INFO:teuthology.orchestra.run.smithi152.stdout:skipping the directory /dev 2023-12-06T03:49:24.229 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:49:25.565 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-06T03:49:25.665 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-06T03:49:25.665 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-06T03:49:25.857 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-06T03:49:25.857 INFO:teuthology.orchestra.run.smithi152.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-06T03:49:26.515 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-06T03:49:26.515 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-12-06T03:49:26.515 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-12-06T03:49:26.515 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-12-06T03:49:26.515 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-12-06T03:49:26.515 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-12-06T03:49:26.515 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-12-06T03:49:26.515 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-12-06T03:49:26.515 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-12-06T03:49:26.515 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-12-06T03:49:26.515 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-12-06T03:49:26.516 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-12-06T03:49:26.517 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-12-06T03:49:26.517 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-12-06T03:49:26.517 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-12-06T03:49:26.518 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-12-06T03:49:26.518 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /sys 2023-12-06T03:49:26.518 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /proc 2023-12-06T03:49:26.518 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /dev 2023-12-06T03:49:26.518 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /run 2023-12-06T03:49:26.518 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /mnt 2023-12-06T03:49:26.518 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /var/tmp 2023-12-06T03:49:26.518 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /home 2023-12-06T03:49:26.518 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /tmp 2023-12-06T03:49:26.518 INFO:teuthology.orchestra.run.smithi132.stdout:skipping the directory /dev 2023-12-06T03:49:26.519 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:49:27.635 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-12-06T03:49:27.635 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:49:27.635 INFO:teuthology.orchestra.run.smithi152.stdout:Upgraded: 2023-12-06T03:49:27.635 INFO:teuthology.orchestra.run.smithi152.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:27.635 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:49:27.635 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:27.635 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:27.635 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:27.635 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:27.635 INFO:teuthology.orchestra.run.smithi152.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-12-06T03:49:27.635 INFO:teuthology.orchestra.run.smithi152.stdout: fmt-6.2.1-1.el8.x86_64 2023-12-06T03:49:27.636 INFO:teuthology.orchestra.run.smithi152.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-06T03:49:27.636 INFO:teuthology.orchestra.run.smithi152.stdout: leveldb-1.22-1.el8.x86_64 2023-12-06T03:49:27.636 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:27.636 INFO:teuthology.orchestra.run.smithi152.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-06T03:49:27.636 INFO:teuthology.orchestra.run.smithi152.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-12-06T03:49:27.636 INFO:teuthology.orchestra.run.smithi152.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-06T03:49:27.636 INFO:teuthology.orchestra.run.smithi152.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:27.636 INFO:teuthology.orchestra.run.smithi152.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-12-06T03:49:27.636 INFO:teuthology.orchestra.run.smithi152.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:27.636 INFO:teuthology.orchestra.run.smithi152.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-06T03:49:27.636 INFO:teuthology.orchestra.run.smithi152.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-06T03:49:27.636 INFO:teuthology.orchestra.run.smithi152.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-12-06T03:49:27.636 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:27.636 INFO:teuthology.orchestra.run.smithi152.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:27.637 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:27.637 INFO:teuthology.orchestra.run.smithi152.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-06T03:49:27.637 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:27.637 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:27.637 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:27.637 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:49:27.637 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:49:27.806 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-test 2023-12-06T03:49:27.847 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-06T03:49:27.937 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-12-06T03:49:27.937 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-06T03:49:28.166 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-12-06T03:49:28.167 INFO:teuthology.orchestra.run.smithi132.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-06T03:49:28.650 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:41 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:49:28.829 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-12-06T03:49:28.829 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-12-06T03:49:28.829 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-12-06T03:49:28.829 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-12-06T03:49:28.829 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-12-06T03:49:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-12-06T03:49:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-12-06T03:49:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-12-06T03:49:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-12-06T03:49:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-12-06T03:49:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-12-06T03:49:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-12-06T03:49:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-12-06T03:49:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-12-06T03:49:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-12-06T03:49:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-12-06T03:49:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-12-06T03:49:28.830 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-12-06T03:49:28.831 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-12-06T03:49:28.831 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-12-06T03:49:28.831 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-12-06T03:49:28.831 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-12-06T03:49:28.831 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-12-06T03:49:28.831 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-12-06T03:49:28.831 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-12-06T03:49:28.831 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-12-06T03:49:28.831 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-12-06T03:49:28.831 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-12-06T03:49:28.831 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-12-06T03:49:30.119 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-12-06T03:49:30.119 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:49:30.119 INFO:teuthology.orchestra.run.smithi132.stdout:Upgraded: 2023-12-06T03:49:30.119 INFO:teuthology.orchestra.run.smithi132.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:30.119 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:49:30.119 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:30.119 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:30.119 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:30.119 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:30.119 INFO:teuthology.orchestra.run.smithi132.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-12-06T03:49:30.119 INFO:teuthology.orchestra.run.smithi132.stdout: fmt-6.2.1-1.el8.x86_64 2023-12-06T03:49:30.119 INFO:teuthology.orchestra.run.smithi132.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-06T03:49:30.119 INFO:teuthology.orchestra.run.smithi132.stdout: leveldb-1.22-1.el8.x86_64 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-06T03:49:30.120 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:30.121 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:30.121 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:30.121 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:49:30.121 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:49:30.349 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-test 2023-12-06T03:49:30.352 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:49:30.353 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:49:30.353 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2023-12-06T03:49:30.353 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:49:30.353 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:49:30.353 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-12-06T03:49:30.354 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2023-12-06T03:49:30.354 INFO:teuthology.orchestra.run.smithi152.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-06T03:49:30.354 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-12-06T03:49:30.354 INFO:teuthology.orchestra.run.smithi152.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-06T03:49:30.354 INFO:teuthology.orchestra.run.smithi152.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-06T03:49:30.354 INFO:teuthology.orchestra.run.smithi152.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-06T03:49:30.354 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:49:30.355 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:49:30.355 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:49:30.355 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2023-12-06T03:49:30.355 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:49:30.356 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 53 M 2023-12-06T03:49:30.356 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 229 M 2023-12-06T03:49:30.356 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:49:30.626 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.2 MB/s | 169 kB 00:00 2023-12-06T03:49:30.651 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 2.0 MB/s | 203 kB 00:00 2023-12-06T03:49:30.702 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-12-06T03:49:30.727 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2023-12-06T03:49:30.778 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 941 kB/s | 71 kB 00:00 2023-12-06T03:49:31.197 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:42 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:49:31.378 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 63 MB/s | 52 MB 00:00 2023-12-06T03:49:31.380 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:49:31.380 INFO:teuthology.orchestra.run.smithi152.stdout:Total 51 MB/s | 53 MB 00:01 2023-12-06T03:49:31.423 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:49:31.444 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:49:31.444 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:49:31.749 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:49:31.752 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:49:32.249 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:49:32.448 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-06T03:49:32.657 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-06T03:49:32.829 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-06T03:49:32.890 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-06T03:49:32.895 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:49:32.896 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:49:32.896 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2023-12-06T03:49:32.897 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:49:32.897 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:49:32.897 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-12-06T03:49:32.897 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T03:49:32.897 INFO:teuthology.orchestra.run.smithi132.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-06T03:49:32.897 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-12-06T03:49:32.897 INFO:teuthology.orchestra.run.smithi132.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-06T03:49:32.897 INFO:teuthology.orchestra.run.smithi132.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-06T03:49:32.897 INFO:teuthology.orchestra.run.smithi132.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-06T03:49:32.897 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:49:32.897 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:49:32.897 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:49:32.897 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2023-12-06T03:49:32.897 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:49:32.898 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 53 M 2023-12-06T03:49:32.898 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 229 M 2023-12-06T03:49:32.898 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:49:33.053 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-06T03:49:33.152 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.8 MB/s | 169 kB 00:00 2023-12-06T03:49:33.178 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 2.3 MB/s | 203 kB 00:00 2023-12-06T03:49:33.212 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-06T03:49:33.228 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 6.3 MB/s | 323 kB 00:00 2023-12-06T03:49:33.253 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2023-12-06T03:49:33.270 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 1.6 MB/s | 71 kB 00:00 2023-12-06T03:49:33.854 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): ceph-test-16.2.4-0.el8.x86_64.rpm 68 MB/s | 52 MB 00:00 2023-12-06T03:49:33.856 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:49:33.856 INFO:teuthology.orchestra.run.smithi132.stdout:Total 55 MB/s | 53 MB 00:00 2023-12-06T03:49:33.899 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:49:33.920 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:49:33.920 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:49:34.234 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:49:34.237 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:49:34.710 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:49:34.924 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-06T03:49:35.107 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-06T03:49:35.255 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-06T03:49:35.333 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-06T03:49:35.503 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-06T03:49:35.671 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-06T03:49:38.100 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-06T03:49:38.326 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-06T03:49:38.610 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-06T03:49:38.610 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-12-06T03:49:38.610 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-12-06T03:49:38.610 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-06T03:49:38.610 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-06T03:49:38.611 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-06T03:49:39.327 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-06T03:49:39.328 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:49:39.328 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:49:39.328 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-12-06T03:49:39.328 INFO:teuthology.orchestra.run.smithi152.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-12-06T03:49:39.328 INFO:teuthology.orchestra.run.smithi152.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-12-06T03:49:39.328 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:49:39.328 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:49:39.554 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph 2023-12-06T03:49:40.416 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:00:53 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:49:40.538 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-12-06T03:49:40.752 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-06T03:49:41.043 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-12-06T03:49:41.043 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-12-06T03:49:41.043 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-12-06T03:49:41.043 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-06T03:49:41.043 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-06T03:49:41.043 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-06T03:49:41.809 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-06T03:49:41.809 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:49:41.809 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:49:41.809 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-12-06T03:49:41.809 INFO:teuthology.orchestra.run.smithi132.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-12-06T03:49:41.809 INFO:teuthology.orchestra.run.smithi132.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-12-06T03:49:41.810 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:49:41.810 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:49:42.017 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph 2023-12-06T03:49:42.092 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:49:42.095 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:49:42.095 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2023-12-06T03:49:42.095 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:49:42.095 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:49:42.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-12-06T03:49:42.095 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2023-12-06T03:49:42.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-12-06T03:49:42.096 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-12-06T03:49:42.096 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-12-06T03:49:42.096 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-12-06T03:49:42.096 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-12-06T03:49:42.096 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-06T03:49:42.096 INFO:teuthology.orchestra.run.smithi152.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-06T03:49:42.096 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-06T03:49:42.096 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-06T03:49:42.096 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-06T03:49:42.096 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-06T03:49:42.096 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-06T03:49:42.096 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-06T03:49:42.096 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-06T03:49:42.096 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-06T03:49:42.097 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-06T03:49:42.098 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-06T03:49:42.098 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:49:42.098 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:49:42.098 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:49:42.098 INFO:teuthology.orchestra.run.smithi152.stdout:Install 32 Packages 2023-12-06T03:49:42.098 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:49:42.098 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 31 M 2023-12-06T03:49:42.099 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 111 M 2023-12-06T03:49:42.099 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:49:42.512 INFO:teuthology.orchestra.run.smithi152.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 120 kB/s | 6.3 kB 00:00 2023-12-06T03:49:42.562 INFO:teuthology.orchestra.run.smithi152.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2023-12-06T03:49:42.654 INFO:teuthology.orchestra.run.smithi152.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 12 MB/s | 2.4 MB 00:00 2023-12-06T03:49:42.688 INFO:teuthology.orchestra.run.smithi152.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2023-12-06T03:49:42.763 INFO:teuthology.orchestra.run.smithi152.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2023-12-06T03:49:42.805 INFO:teuthology.orchestra.run.smithi152.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 934 kB/s | 39 kB 00:00 2023-12-06T03:49:42.830 INFO:teuthology.orchestra.run.smithi152.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.1 MB/s | 157 kB 00:00 2023-12-06T03:49:42.864 INFO:teuthology.orchestra.run.smithi152.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.7 MB/s | 103 kB 00:00 2023-12-06T03:49:42.876 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:00:53 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:49:42.889 INFO:teuthology.orchestra.run.smithi152.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 974 kB/s | 57 kB 00:00 2023-12-06T03:49:43.089 INFO:teuthology.orchestra.run.smithi152.stdout:(10/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 34 MB/s | 18 MB 00:00 2023-12-06T03:49:43.123 INFO:teuthology.orchestra.run.smithi152.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 1.0 MB/s | 249 kB 00:00 2023-12-06T03:49:43.165 INFO:teuthology.orchestra.run.smithi152.stdout:(12/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.5 MB/s | 457 kB 00:00 2023-12-06T03:49:43.190 INFO:teuthology.orchestra.run.smithi152.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 1.6 MB/s | 109 kB 00:00 2023-12-06T03:49:43.240 INFO:teuthology.orchestra.run.smithi152.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 4.2 MB/s | 641 kB 00:00 2023-12-06T03:49:43.499 INFO:teuthology.orchestra.run.smithi152.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 131 kB/s | 44 kB 00:00 2023-12-06T03:49:43.599 INFO:teuthology.orchestra.run.smithi152.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 453 kB/s | 185 kB 00:00 2023-12-06T03:49:43.649 INFO:teuthology.orchestra.run.smithi152.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 422 kB/s | 173 kB 00:00 2023-12-06T03:49:43.700 INFO:teuthology.orchestra.run.smithi152.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2023-12-06T03:49:43.750 INFO:teuthology.orchestra.run.smithi152.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 175 kB/s | 18 kB 00:00 2023-12-06T03:49:43.800 INFO:teuthology.orchestra.run.smithi152.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 486 kB/s | 49 kB 00:00 2023-12-06T03:49:43.834 INFO:teuthology.orchestra.run.smithi152.stdout:(21/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.1 MB/s | 384 kB 00:00 2023-12-06T03:49:43.859 INFO:teuthology.orchestra.run.smithi152.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 539 kB/s | 59 kB 00:00 2023-12-06T03:49:43.935 INFO:teuthology.orchestra.run.smithi152.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 155 kB/s | 16 kB 00:00 2023-12-06T03:49:43.960 INFO:teuthology.orchestra.run.smithi152.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 186 kB/s | 19 kB 00:00 2023-12-06T03:49:43.985 INFO:teuthology.orchestra.run.smithi152.stdout:(25/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.5 MB/s | 283 kB 00:00 2023-12-06T03:49:44.035 INFO:teuthology.orchestra.run.smithi152.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 238 kB/s | 24 kB 00:00 2023-12-06T03:49:44.061 INFO:teuthology.orchestra.run.smithi152.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 291 kB/s | 29 kB 00:00 2023-12-06T03:49:44.086 INFO:teuthology.orchestra.run.smithi152.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 273 kB/s | 27 kB 00:00 2023-12-06T03:49:44.161 INFO:teuthology.orchestra.run.smithi152.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2023-12-06T03:49:44.186 INFO:teuthology.orchestra.run.smithi152.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 853 kB/s | 85 kB 00:00 2023-12-06T03:49:44.262 INFO:teuthology.orchestra.run.smithi152.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.2 MB/s | 251 kB 00:00 2023-12-06T03:49:44.287 INFO:teuthology.orchestra.run.smithi152.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 183 kB/s | 23 kB 00:00 2023-12-06T03:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:49:44.288 INFO:teuthology.orchestra.run.smithi152.stdout:Total 14 MB/s | 31 MB 00:02 2023-12-06T03:49:44.373 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:49:44.411 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:49:44.411 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:49:44.547 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:49:44.550 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:49:44.551 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T03:49:44.551 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:49:44.551 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:49:44.551 INFO:teuthology.orchestra.run.smithi132.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-12-06T03:49:44.551 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T03:49:44.551 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-12-06T03:49:44.551 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-12-06T03:49:44.551 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-12-06T03:49:44.552 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-12-06T03:49:44.552 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-12-06T03:49:44.552 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-06T03:49:44.552 INFO:teuthology.orchestra.run.smithi132.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-06T03:49:44.552 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-06T03:49:44.552 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-06T03:49:44.552 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-06T03:49:44.553 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-06T03:49:44.553 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-06T03:49:44.553 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-06T03:49:44.553 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-06T03:49:44.553 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-06T03:49:44.553 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-06T03:49:44.553 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-06T03:49:44.553 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-06T03:49:44.553 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-06T03:49:44.554 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-06T03:49:44.554 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-06T03:49:44.554 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-06T03:49:44.554 INFO:teuthology.orchestra.run.smithi132.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-06T03:49:44.554 INFO:teuthology.orchestra.run.smithi132.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-06T03:49:44.554 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-06T03:49:44.554 INFO:teuthology.orchestra.run.smithi132.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-06T03:49:44.554 INFO:teuthology.orchestra.run.smithi132.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-06T03:49:44.554 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-06T03:49:44.554 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-06T03:49:44.554 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-06T03:49:44.554 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-06T03:49:44.555 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:49:44.555 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:49:44.555 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:49:44.555 INFO:teuthology.orchestra.run.smithi132.stdout:Install 32 Packages 2023-12-06T03:49:44.555 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:49:44.556 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 31 M 2023-12-06T03:49:44.556 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 111 M 2023-12-06T03:49:44.556 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:49:44.728 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:49:44.731 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:49:44.881 INFO:teuthology.orchestra.run.smithi132.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 158 kB/s | 6.3 kB 00:00 2023-12-06T03:49:44.931 INFO:teuthology.orchestra.run.smithi132.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2023-12-06T03:49:44.990 INFO:teuthology.orchestra.run.smithi132.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2023-12-06T03:49:45.015 INFO:teuthology.orchestra.run.smithi132.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2023-12-06T03:49:45.099 INFO:teuthology.orchestra.run.smithi132.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 23 MB/s | 5.0 MB 00:00 2023-12-06T03:49:45.124 INFO:teuthology.orchestra.run.smithi132.stdout:(6/32): python3-mako-1.0.6-14.el8.noarch.rpm 1.4 MB/s | 157 kB 00:00 2023-12-06T03:49:45.158 INFO:teuthology.orchestra.run.smithi132.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 3.0 MB/s | 103 kB 00:00 2023-12-06T03:49:45.175 INFO:teuthology.orchestra.run.smithi132.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 514 kB/s | 39 kB 00:00 2023-12-06T03:49:45.191 INFO:teuthology.orchestra.run.smithi132.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 1.7 MB/s | 57 kB 00:00 2023-12-06T03:49:45.242 INFO:teuthology.orchestra.run.smithi132.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 6.7 MB/s | 457 kB 00:00 2023-12-06T03:49:45.278 INFO:teuthology.orchestra.run.smithi132.stdout:(11/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 2.8 MB/s | 249 kB 00:00 2023-12-06T03:49:45.520 INFO:teuthology.orchestra.run.smithi132.stdout:(12/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 31 MB/s | 18 MB 00:00 2023-12-06T03:49:45.545 INFO:teuthology.orchestra.run.smithi132.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 408 kB/s | 109 kB 00:00 2023-12-06T03:49:45.570 INFO:teuthology.orchestra.run.smithi132.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 1.9 MB/s | 641 kB 00:00 2023-12-06T03:49:45.679 INFO:teuthology.orchestra.run.smithi132.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 275 kB/s | 44 kB 00:00 2023-12-06T03:49:45.737 INFO:teuthology.orchestra.run.smithi132.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 964 kB/s | 185 kB 00:00 2023-12-06T03:49:45.763 INFO:teuthology.orchestra.run.smithi132.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 942 kB/s | 173 kB 00:00 2023-12-06T03:49:45.788 INFO:teuthology.orchestra.run.smithi132.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 221 kB/s | 11 kB 00:00 2023-12-06T03:49:45.813 INFO:teuthology.orchestra.run.smithi132.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 351 kB/s | 18 kB 00:00 2023-12-06T03:49:45.836 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:49:45.838 INFO:teuthology.orchestra.run.smithi132.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 974 kB/s | 49 kB 00:00 2023-12-06T03:49:45.864 INFO:teuthology.orchestra.run.smithi132.stdout:(21/32): python3-cherrypy-18.4.0-1.el8.noarch.r 2.0 MB/s | 384 kB 00:00 2023-12-06T03:49:45.889 INFO:teuthology.orchestra.run.smithi132.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 777 kB/s | 59 kB 00:00 2023-12-06T03:49:45.914 INFO:teuthology.orchestra.run.smithi132.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2023-12-06T03:49:45.940 INFO:teuthology.orchestra.run.smithi132.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 373 kB/s | 19 kB 00:00 2023-12-06T03:49:45.965 INFO:teuthology.orchestra.run.smithi132.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 476 kB/s | 24 kB 00:00 2023-12-06T03:49:45.990 INFO:teuthology.orchestra.run.smithi132.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 583 kB/s | 29 kB 00:00 2023-12-06T03:49:46.015 INFO:teuthology.orchestra.run.smithi132.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 550 kB/s | 27 kB 00:00 2023-12-06T03:49:46.031 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-06T03:49:46.041 INFO:teuthology.orchestra.run.smithi132.stdout:(28/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.4 MB/s | 283 kB 00:00 2023-12-06T03:49:46.099 INFO:teuthology.orchestra.run.smithi132.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.3 MB/s | 255 kB 00:00 2023-12-06T03:49:46.125 INFO:teuthology.orchestra.run.smithi132.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2023-12-06T03:49:46.142 INFO:teuthology.orchestra.run.smithi132.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 548 kB/s | 23 kB 00:00 2023-12-06T03:49:46.167 INFO:teuthology.orchestra.run.smithi132.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.6 MB/s | 251 kB 00:00 2023-12-06T03:49:46.171 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:49:46.171 INFO:teuthology.orchestra.run.smithi132.stdout:Total 19 MB/s | 31 MB 00:01 2023-12-06T03:49:46.226 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-06T03:49:46.254 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:49:46.293 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:49:46.293 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:49:46.467 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-06T03:49:46.609 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:49:46.612 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:49:46.650 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-06T03:49:46.806 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-06T03:49:46.969 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-06T03:49:47.113 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-06T03:49:47.307 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-06T03:49:47.488 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-06T03:49:47.684 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-06T03:49:47.694 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:49:47.897 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-06T03:49:47.900 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-06T03:49:48.093 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-06T03:49:48.153 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-06T03:49:48.335 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-06T03:49:48.507 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-06T03:49:48.542 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-06T03:49:48.730 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-06T03:49:48.804 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-06T03:49:48.903 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-06T03:49:48.959 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-06T03:49:49.063 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-06T03:49:49.192 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-06T03:49:49.225 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-06T03:49:49.352 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-06T03:49:49.400 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-06T03:49:49.620 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-06T03:49:49.678 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-06T03:49:49.834 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-06T03:49:49.890 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-06T03:49:50.074 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-06T03:49:50.078 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-06T03:49:50.282 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-06T03:49:50.431 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-06T03:49:50.571 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-06T03:49:50.672 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-06T03:49:50.824 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-06T03:49:50.835 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-06T03:49:51.038 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-06T03:49:51.093 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-06T03:49:51.278 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-06T03:49:51.404 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-06T03:49:51.562 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-06T03:49:51.682 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-06T03:49:51.741 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-06T03:49:51.913 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-06T03:49:52.083 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-12-06T03:49:52.146 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-06T03:49:52.286 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-06T03:49:52.392 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-06T03:49:52.629 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-06T03:49:52.853 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-06T03:49:53.201 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-06T03:49:53.500 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-06T03:49:53.817 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-06T03:49:53.891 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-12-06T03:49:54.019 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-06T03:49:54.093 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-06T03:49:54.453 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-06T03:49:54.677 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-06T03:49:54.901 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-06T03:49:55.117 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-06T03:49:55.135 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-06T03:49:55.267 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-06T03:49:55.624 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-12-06T03:49:55.657 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-06T03:49:55.658 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-12-06T03:49:55.658 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-12-06T03:49:55.658 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-12-06T03:49:55.658 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-12-06T03:49:55.658 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-12-06T03:49:55.658 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-12-06T03:49:55.658 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-06T03:49:55.658 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-06T03:49:55.658 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-06T03:49:55.658 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-06T03:49:55.658 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-06T03:49:55.658 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-06T03:49:55.659 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-06T03:49:55.659 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-06T03:49:55.659 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-06T03:49:55.659 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-06T03:49:55.659 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-06T03:49:55.659 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-06T03:49:55.659 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-06T03:49:55.659 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-06T03:49:55.659 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-06T03:49:55.659 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-06T03:49:55.659 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-06T03:49:55.659 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-06T03:49:55.659 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-06T03:49:55.659 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-06T03:49:55.660 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-06T03:49:55.660 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-06T03:49:55.660 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-06T03:49:55.660 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-06T03:49:55.660 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-06T03:49:55.810 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-06T03:49:56.245 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-12-06T03:49:56.493 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-06T03:49:56.715 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-12-06T03:49:56.861 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-06T03:49:56.861 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:49:56.861 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:49:56.861 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:56.861 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:56.861 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:56.861 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-12-06T03:49:56.861 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:56.861 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:56.861 INFO:teuthology.orchestra.run.smithi152.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-06T03:49:56.861 INFO:teuthology.orchestra.run.smithi152.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-06T03:49:56.861 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-06T03:49:56.861 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-06T03:49:56.862 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-06T03:49:56.862 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-06T03:49:56.862 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-06T03:49:56.862 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-06T03:49:56.862 INFO:teuthology.orchestra.run.smithi152.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-06T03:49:56.862 INFO:teuthology.orchestra.run.smithi152.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-06T03:49:56.862 INFO:teuthology.orchestra.run.smithi152.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-06T03:49:56.862 INFO:teuthology.orchestra.run.smithi152.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-06T03:49:56.862 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-06T03:49:56.862 INFO:teuthology.orchestra.run.smithi152.stdout: python3-portend-2.6-1.el8.noarch 2023-12-06T03:49:56.862 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-06T03:49:56.862 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-06T03:49:56.862 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-06T03:49:56.862 INFO:teuthology.orchestra.run.smithi152.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-06T03:49:56.863 INFO:teuthology.orchestra.run.smithi152.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-06T03:49:56.863 INFO:teuthology.orchestra.run.smithi152.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-06T03:49:56.863 INFO:teuthology.orchestra.run.smithi152.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-06T03:49:56.863 INFO:teuthology.orchestra.run.smithi152.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-06T03:49:56.863 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-06T03:49:56.863 INFO:teuthology.orchestra.run.smithi152.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-06T03:49:56.863 INFO:teuthology.orchestra.run.smithi152.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-06T03:49:56.863 INFO:teuthology.orchestra.run.smithi152.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-06T03:49:56.863 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:49:56.863 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:49:56.927 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-06T03:49:56.945 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-12-06T03:49:57.071 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-base 2023-12-06T03:49:57.085 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-06T03:49:57.505 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-12-06T03:49:57.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-12-06T03:49:57.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-12-06T03:49:57.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-12-06T03:49:57.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-12-06T03:49:57.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-12-06T03:49:57.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-12-06T03:49:57.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-06T03:49:57.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-06T03:49:57.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-06T03:49:57.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-06T03:49:57.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-06T03:49:57.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-06T03:49:57.506 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-06T03:49:57.507 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-06T03:49:57.507 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-06T03:49:57.507 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-06T03:49:57.507 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-06T03:49:57.507 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-06T03:49:57.507 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-06T03:49:57.507 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-06T03:49:57.507 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-06T03:49:57.507 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-06T03:49:57.507 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-06T03:49:57.507 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-06T03:49:57.507 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-06T03:49:57.507 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-06T03:49:57.508 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-06T03:49:57.508 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-06T03:49:57.509 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-06T03:49:57.509 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-06T03:49:57.509 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-06T03:49:57.933 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:10 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:49:58.610 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-06T03:49:58.610 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:49:58.610 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:49:58.610 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:58.610 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:58.610 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:58.610 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-12-06T03:49:58.611 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:58.611 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-12-06T03:49:58.611 INFO:teuthology.orchestra.run.smithi132.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-06T03:49:58.611 INFO:teuthology.orchestra.run.smithi132.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-06T03:49:58.611 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-06T03:49:58.611 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-06T03:49:58.611 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-06T03:49:58.611 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-06T03:49:58.611 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-06T03:49:58.611 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-06T03:49:58.611 INFO:teuthology.orchestra.run.smithi132.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-06T03:49:58.611 INFO:teuthology.orchestra.run.smithi132.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-06T03:49:58.611 INFO:teuthology.orchestra.run.smithi132.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-06T03:49:58.611 INFO:teuthology.orchestra.run.smithi132.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: python3-portend-2.6-1.el8.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-06T03:49:58.612 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:49:58.613 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:49:58.813 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-base 2023-12-06T03:49:59.386 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:49:59.613 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:49:59.614 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2023-12-06T03:49:59.614 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:49:59.675 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:10 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:50:00.006 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install cephadm 2023-12-06T03:50:00.864 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:13 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:50:01.124 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:50:01.348 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:50:01.349 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T03:50:01.349 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:50:01.459 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install cephadm 2023-12-06T03:50:02.321 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:13 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:50:02.535 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:50:02.536 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:02.536 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2023-12-06T03:50:02.536 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:02.536 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:50:02.536 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-12-06T03:50:02.536 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:02.536 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:50:02.536 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:02.536 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2023-12-06T03:50:02.536 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:02.537 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 71 k 2023-12-06T03:50:02.537 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 290 k 2023-12-06T03:50:02.537 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:50:02.581 INFO:teuthology.orchestra.run.smithi152.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 71 kB 00:00 2023-12-06T03:50:02.581 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:02.581 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.6 MB/s | 71 kB 00:00 2023-12-06T03:50:02.581 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:50:02.588 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:50:02.589 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:50:02.590 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:50:02.593 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:50:02.633 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:50:03.152 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-06T03:50:03.338 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-06T03:50:03.564 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-06T03:50:03.991 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:50:03.991 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:03.991 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2023-12-06T03:50:03.991 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:03.991 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:50:03.992 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-12-06T03:50:03.992 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:03.992 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:50:03.992 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:03.992 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T03:50:03.992 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:03.992 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 71 k 2023-12-06T03:50:03.993 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 290 k 2023-12-06T03:50:03.993 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:50:04.030 INFO:teuthology.orchestra.run.smithi132.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.9 MB/s | 71 kB 00:00 2023-12-06T03:50:04.030 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:04.031 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.8 MB/s | 71 kB 00:00 2023-12-06T03:50:04.031 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:50:04.038 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:50:04.039 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:50:04.039 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:50:04.043 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:50:04.083 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:50:04.206 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-06T03:50:04.206 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:04.206 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:50:04.206 INFO:teuthology.orchestra.run.smithi152.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-12-06T03:50:04.206 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:04.206 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:50:04.397 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-immutable-object-cache 2023-12-06T03:50:04.565 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-06T03:50:04.788 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-06T03:50:05.016 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-06T03:50:05.255 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:18 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:50:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-12-06T03:50:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:05.731 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:50:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-12-06T03:50:05.731 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:05.731 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:50:05.944 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-immutable-object-cache 2023-12-06T03:50:06.814 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:17 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:50:06.944 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:50:06.944 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:06.944 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repo Size 2023-12-06T03:50:06.944 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:06.944 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:50:06.944 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-12-06T03:50:06.944 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:06.945 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:50:06.945 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:06.945 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2023-12-06T03:50:06.945 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:06.945 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 163 k 2023-12-06T03:50:06.945 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 587 k 2023-12-06T03:50:06.945 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:50:06.993 INFO:teuthology.orchestra.run.smithi152.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.4 MB/s | 163 kB 00:00 2023-12-06T03:50:06.993 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:06.993 INFO:teuthology.orchestra.run.smithi152.stdout:Total 3.3 MB/s | 163 kB 00:00 2023-12-06T03:50:06.994 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:50:07.004 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:50:07.004 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:50:07.052 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:50:07.054 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:50:07.164 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:50:07.324 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:50:07.728 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:50:08.382 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:50:08.383 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:08.383 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:50:08.383 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-12-06T03:50:08.383 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:08.383 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:50:08.507 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:50:08.508 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:08.508 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repo Size 2023-12-06T03:50:08.508 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:08.508 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:50:08.508 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-12-06T03:50:08.508 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:08.508 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:50:08.508 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:08.508 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T03:50:08.508 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:08.509 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 163 k 2023-12-06T03:50:08.509 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 587 k 2023-12-06T03:50:08.509 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:50:08.552 INFO:teuthology.orchestra.run.smithi132.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.7 MB/s | 163 kB 00:00 2023-12-06T03:50:08.553 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:08.553 INFO:teuthology.orchestra.run.smithi132.stdout:Total 3.6 MB/s | 163 kB 00:00 2023-12-06T03:50:08.553 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:50:08.564 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:50:08.564 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:50:08.611 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:50:08.614 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:50:08.617 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr 2023-12-06T03:50:08.722 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:50:08.891 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:50:09.307 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:50:09.478 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:22 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:50:10.000 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:50:10.000 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:10.000 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:50:10.000 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-12-06T03:50:10.000 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:10.000 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:50:10.210 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr 2023-12-06T03:50:10.929 INFO:teuthology.orchestra.run.smithi152.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:50:11.072 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:22 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:50:11.153 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:50:11.154 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2023-12-06T03:50:11.155 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:50:11.252 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-dashboard 2023-12-06T03:50:12.104 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:25 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:50:12.508 INFO:teuthology.orchestra.run.smithi132.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:50:12.731 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:50:12.732 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T03:50:12.732 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:50:12.832 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-dashboard 2023-12-06T03:50:13.703 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:24 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:50:13.797 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:50:13.798 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:13.798 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2023-12-06T03:50:13.798 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:13.798 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:50:13.798 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-12-06T03:50:13.799 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2023-12-06T03:50:13.799 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-12-06T03:50:13.799 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-12-06T03:50:13.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-06T03:50:13.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-06T03:50:13.799 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-06T03:50:13.799 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:13.799 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:50:13.799 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:13.799 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2023-12-06T03:50:13.799 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:13.800 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 3.0 M 2023-12-06T03:50:13.800 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 15 M 2023-12-06T03:50:13.800 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:50:14.103 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 407 kB/s | 24 kB 00:00 2023-12-06T03:50:14.120 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 118 kB/s | 8.9 kB 00:00 2023-12-06T03:50:14.187 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 19 MB/s | 2.7 MB 00:00 2023-12-06T03:50:14.238 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 318 kB/s | 43 kB 00:00 2023-12-06T03:50:15.113 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 34 kB/s | 34 kB 00:00 2023-12-06T03:50:15.407 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:50:15.408 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:15.408 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T03:50:15.409 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:15.409 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:50:15.409 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-12-06T03:50:15.409 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T03:50:15.409 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-12-06T03:50:15.409 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-12-06T03:50:15.409 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-06T03:50:15.409 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-06T03:50:15.409 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-06T03:50:15.409 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:15.409 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:50:15.409 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:15.409 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2023-12-06T03:50:15.409 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:15.410 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 3.0 M 2023-12-06T03:50:15.410 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 15 M 2023-12-06T03:50:15.410 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:50:15.580 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 141 kB/s | 196 kB 00:01 2023-12-06T03:50:15.583 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:15.583 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.7 MB/s | 3.0 MB 00:01 2023-12-06T03:50:15.595 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:50:15.605 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:50:15.605 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:50:15.764 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:50:15.767 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:50:15.796 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 524 kB/s | 24 kB 00:00 2023-12-06T03:50:15.813 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 142 kB/s | 8.9 kB 00:00 2023-12-06T03:50:15.872 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 22 MB/s | 2.7 MB 00:00 2023-12-06T03:50:15.914 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 363 kB/s | 43 kB 00:00 2023-12-06T03:50:16.040 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:50:16.064 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 134 kB/s | 34 kB 00:00 2023-12-06T03:50:16.277 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-06T03:50:16.281 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 481 kB/s | 196 kB 00:00 2023-12-06T03:50:16.283 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:16.283 INFO:teuthology.orchestra.run.smithi132.stdout:Total 3.4 MB/s | 3.0 MB 00:00 2023-12-06T03:50:16.295 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:50:16.303 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:50:16.303 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:50:16.478 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:50:16.481 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:50:16.522 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-06T03:50:16.732 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-06T03:50:16.748 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:50:16.878 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-12-06T03:50:16.954 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-06T03:50:17.224 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-06T03:50:17.425 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-06T03:50:17.546 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-12-06T03:50:17.843 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-12-06T03:50:18.107 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-06T03:50:18.398 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-06T03:50:18.399 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-12-06T03:50:18.399 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-12-06T03:50:18.399 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-12-06T03:50:18.399 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-06T03:50:18.399 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-06T03:50:18.503 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-12-06T03:50:18.748 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-06T03:50:19.030 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-12-06T03:50:19.030 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-12-06T03:50:19.030 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-12-06T03:50:19.030 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-12-06T03:50:19.030 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-06T03:50:19.030 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-06T03:50:19.094 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-06T03:50:19.094 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:19.094 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:50:19.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-12-06T03:50:19.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-12-06T03:50:19.095 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-12-06T03:50:19.095 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-06T03:50:19.095 INFO:teuthology.orchestra.run.smithi152.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-06T03:50:19.095 INFO:teuthology.orchestra.run.smithi152.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-06T03:50:19.095 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:19.095 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:50:19.243 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-06T03:50:19.734 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-06T03:50:19.735 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:19.735 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:50:19.735 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-12-06T03:50:19.735 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-12-06T03:50:19.735 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-12-06T03:50:19.735 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-06T03:50:19.735 INFO:teuthology.orchestra.run.smithi132.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-06T03:50:19.735 INFO:teuthology.orchestra.run.smithi132.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-06T03:50:19.735 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:19.735 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:50:19.849 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-06T03:50:20.091 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:33 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:50:20.701 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:31 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:50:21.766 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:50:21.767 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================================ 2023-12-06T03:50:21.768 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2023-12-06T03:50:21.768 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================================ 2023-12-06T03:50:21.768 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:50:21.768 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-12-06T03:50:21.768 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2023-12-06T03:50:21.768 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-06T03:50:21.768 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-06T03:50:21.768 INFO:teuthology.orchestra.run.smithi152.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-06T03:50:21.768 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-06T03:50:21.768 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-06T03:50:21.768 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-06T03:50:21.768 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-12-06T03:50:21.769 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:21.769 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:50:21.769 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================================ 2023-12-06T03:50:21.769 INFO:teuthology.orchestra.run.smithi152.stdout:Install 8 Packages 2023-12-06T03:50:21.769 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:21.769 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 37 M 2023-12-06T03:50:21.770 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 210 M 2023-12-06T03:50:21.770 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:50:22.159 INFO:teuthology.orchestra.run.smithi152.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 57 MB/s | 8.6 MB 00:00 2023-12-06T03:50:22.372 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:50:22.373 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================================ 2023-12-06T03:50:22.373 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T03:50:22.373 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================================ 2023-12-06T03:50:22.373 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:50:22.373 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-12-06T03:50:22.374 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T03:50:22.374 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-06T03:50:22.374 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-06T03:50:22.374 INFO:teuthology.orchestra.run.smithi132.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-06T03:50:22.374 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-06T03:50:22.374 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-06T03:50:22.374 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-06T03:50:22.375 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-12-06T03:50:22.375 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:22.375 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:50:22.375 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================================ 2023-12-06T03:50:22.375 INFO:teuthology.orchestra.run.smithi132.stdout:Install 8 Packages 2023-12-06T03:50:22.375 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:22.376 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 37 M 2023-12-06T03:50:22.376 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 210 M 2023-12-06T03:50:22.376 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:50:22.455 INFO:teuthology.orchestra.run.smithi152.stdout:(2/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 13 MB/s | 3.7 MB 00:00 2023-12-06T03:50:22.639 INFO:teuthology.orchestra.run.smithi152.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 7.7 MB/s | 4.9 MB 00:00 2023-12-06T03:50:22.653 INFO:teuthology.orchestra.run.smithi132.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 45 MB/s | 8.6 MB 00:00 2023-12-06T03:50:22.662 INFO:teuthology.orchestra.run.smithi152.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.1 MB/s | 225 kB 00:00 2023-12-06T03:50:22.721 INFO:teuthology.orchestra.run.smithi152.stdout:(5/8): openblas-0.3.15-4.el8.x86_64.rpm 6.6 MB/s | 4.7 MB 00:00 2023-12-06T03:50:22.770 INFO:teuthology.orchestra.run.smithi132.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 16 MB/s | 4.9 MB 00:00 2023-12-06T03:50:22.804 INFO:teuthology.orchestra.run.smithi132.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 6.6 MB/s | 225 kB 00:00 2023-12-06T03:50:22.821 INFO:teuthology.orchestra.run.smithi152.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 1.7 MB/s | 172 kB 00:00 2023-12-06T03:50:22.879 INFO:teuthology.orchestra.run.smithi152.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.9 MB/s | 645 kB 00:00 2023-12-06T03:50:22.896 INFO:teuthology.orchestra.run.smithi132.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 15 MB/s | 3.7 MB 00:00 2023-12-06T03:50:22.955 INFO:teuthology.orchestra.run.smithi132.stdout:(5/8): openblas-0.3.15-4.el8.x86_64.rpm 9.6 MB/s | 4.7 MB 00:00 2023-12-06T03:50:23.022 INFO:teuthology.orchestra.run.smithi152.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 36 MB/s | 14 MB 00:00 2023-12-06T03:50:23.022 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:23.022 INFO:teuthology.orchestra.run.smithi152.stdout:Total 29 MB/s | 37 MB 00:01 2023-12-06T03:50:23.122 INFO:teuthology.orchestra.run.smithi132.stdout:(6/8): python3-scipy-1.0.0-21.module_el8.5.0+77 43 MB/s | 14 MB 00:00 2023-12-06T03:50:23.147 INFO:teuthology.orchestra.run.smithi132.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 899 kB/s | 172 kB 00:00 2023-12-06T03:50:23.205 INFO:teuthology.orchestra.run.smithi132.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 2.0 MB/s | 645 kB 00:00 2023-12-06T03:50:23.206 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:23.206 INFO:teuthology.orchestra.run.smithi132.stdout:Total 44 MB/s | 37 MB 00:00 2023-12-06T03:50:23.255 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:50:23.273 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:50:23.273 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:50:23.439 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:50:23.457 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:50:23.457 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:50:23.593 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:50:23.595 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:50:23.777 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:50:23.780 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:50:24.214 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:50:24.346 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-06T03:50:24.375 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:50:24.499 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-06T03:50:24.588 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-06T03:50:24.679 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-06T03:50:24.909 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-06T03:50:25.285 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-06T03:50:25.489 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-06T03:50:25.547 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-06T03:50:26.098 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-06T03:50:26.153 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-06T03:50:26.270 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-06T03:50:26.338 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-06T03:50:26.952 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-06T03:50:26.955 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-06T03:50:27.146 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-06T03:50:27.324 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-06T03:50:27.569 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-06T03:50:27.838 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-06T03:50:28.140 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-06T03:50:28.388 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-06T03:50:29.326 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-06T03:50:30.143 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-06T03:50:30.637 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-12-06T03:50:30.833 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-06T03:50:31.147 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-06T03:50:31.148 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-12-06T03:50:31.148 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-06T03:50:31.148 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-06T03:50:31.148 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-06T03:50:31.148 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-06T03:50:31.148 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-12-06T03:50:31.148 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-06T03:50:31.448 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-12-06T03:50:31.654 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-06T03:50:31.859 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-06T03:50:31.860 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:31.860 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:50:31.860 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-12-06T03:50:31.860 INFO:teuthology.orchestra.run.smithi152.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-06T03:50:31.860 INFO:teuthology.orchestra.run.smithi152.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-06T03:50:31.860 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-06T03:50:31.860 INFO:teuthology.orchestra.run.smithi152.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-06T03:50:31.860 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-06T03:50:31.860 INFO:teuthology.orchestra.run.smithi152.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-06T03:50:31.860 INFO:teuthology.orchestra.run.smithi152.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-12-06T03:50:31.860 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:31.860 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:50:31.958 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-12-06T03:50:31.958 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-12-06T03:50:31.958 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-06T03:50:31.958 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-06T03:50:31.958 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-06T03:50:31.959 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-06T03:50:31.959 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-12-06T03:50:31.959 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-06T03:50:31.967 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-rook 2023-12-06T03:50:32.674 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-06T03:50:32.674 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:32.674 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:50:32.674 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-12-06T03:50:32.674 INFO:teuthology.orchestra.run.smithi132.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-06T03:50:32.674 INFO:teuthology.orchestra.run.smithi132.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-06T03:50:32.675 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-06T03:50:32.675 INFO:teuthology.orchestra.run.smithi132.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-06T03:50:32.675 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-06T03:50:32.675 INFO:teuthology.orchestra.run.smithi132.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-06T03:50:32.675 INFO:teuthology.orchestra.run.smithi132.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-12-06T03:50:32.675 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:32.675 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:50:32.797 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-rook 2023-12-06T03:50:32.821 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:45 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:50:33.654 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:44 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:50:34.515 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:50:34.517 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:34.517 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2023-12-06T03:50:34.517 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:34.517 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:50:34.517 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-12-06T03:50:34.517 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2023-12-06T03:50:34.517 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-06T03:50:34.517 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-06T03:50:34.517 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-06T03:50:34.517 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-06T03:50:34.517 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-06T03:50:34.517 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-06T03:50:34.518 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-06T03:50:34.518 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-06T03:50:34.518 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-06T03:50:34.518 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-06T03:50:34.518 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-06T03:50:34.518 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-06T03:50:34.518 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:34.518 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:50:34.518 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:34.518 INFO:teuthology.orchestra.run.smithi152.stdout:Install 13 Packages 2023-12-06T03:50:34.518 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:34.519 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 1.9 M 2023-12-06T03:50:34.519 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 24 M 2023-12-06T03:50:34.519 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:50:34.984 INFO:teuthology.orchestra.run.smithi152.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 726 kB/s | 27 kB 00:00 2023-12-06T03:50:35.010 INFO:teuthology.orchestra.run.smithi152.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 428 kB/s | 27 kB 00:00 2023-12-06T03:50:35.035 INFO:teuthology.orchestra.run.smithi152.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 229 kB/s | 20 kB 00:00 2023-12-06T03:50:35.085 INFO:teuthology.orchestra.run.smithi152.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2023-12-06T03:50:35.111 INFO:teuthology.orchestra.run.smithi152.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2023-12-06T03:50:35.136 INFO:teuthology.orchestra.run.smithi152.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.5 MB/s | 155 kB 00:00 2023-12-06T03:50:35.161 INFO:teuthology.orchestra.run.smithi152.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 564 kB/s | 43 kB 00:00 2023-12-06T03:50:35.336 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:50:35.338 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:35.338 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T03:50:35.338 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:35.338 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:50:35.338 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-12-06T03:50:35.339 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T03:50:35.339 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-06T03:50:35.339 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-06T03:50:35.339 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-06T03:50:35.339 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-06T03:50:35.339 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-06T03:50:35.339 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-06T03:50:35.339 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-06T03:50:35.339 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-06T03:50:35.339 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-06T03:50:35.339 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-06T03:50:35.339 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-06T03:50:35.339 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-06T03:50:35.339 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:35.340 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:50:35.340 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:35.340 INFO:teuthology.orchestra.run.smithi132.stdout:Install 13 Packages 2023-12-06T03:50:35.340 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:35.340 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 1.9 M 2023-12-06T03:50:35.340 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 24 M 2023-12-06T03:50:35.340 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:50:35.836 INFO:teuthology.orchestra.run.smithi152.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 23 kB/s | 16 kB 00:00 2023-12-06T03:50:36.062 INFO:teuthology.orchestra.run.smithi152.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 35 kB/s | 33 kB 00:00 2023-12-06T03:50:36.072 INFO:teuthology.orchestra.run.smithi132.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 670 kB/s | 27 kB 00:00 2023-12-06T03:50:36.098 INFO:teuthology.orchestra.run.smithi132.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 407 kB/s | 27 kB 00:00 2023-12-06T03:50:36.123 INFO:teuthology.orchestra.run.smithi132.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 221 kB/s | 20 kB 00:00 2023-12-06T03:50:36.173 INFO:teuthology.orchestra.run.smithi132.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.1 MB/s | 110 kB 00:00 2023-12-06T03:50:36.199 INFO:teuthology.orchestra.run.smithi132.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 1.0 MB/s | 126 kB 00:00 2023-12-06T03:50:36.224 INFO:teuthology.orchestra.run.smithi132.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 1.5 MB/s | 155 kB 00:00 2023-12-06T03:50:36.258 INFO:teuthology.orchestra.run.smithi132.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 508 kB/s | 43 kB 00:00 2023-12-06T03:50:36.295 INFO:teuthology.orchestra.run.smithi152.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 72 kB/s | 81 kB 00:01 2023-12-06T03:50:36.554 INFO:teuthology.orchestra.run.smithi152.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 131 kB/s | 64 kB 00:00 2023-12-06T03:50:36.612 INFO:teuthology.orchestra.run.smithi152.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 193 kB/s | 61 kB 00:00 2023-12-06T03:50:36.683 INFO:teuthology.orchestra.run.smithi132.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 68 kB/s | 33 kB 00:00 2023-12-06T03:50:36.708 INFO:teuthology.orchestra.run.smithi132.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 34 kB/s | 16 kB 00:00 2023-12-06T03:50:36.767 INFO:teuthology.orchestra.run.smithi132.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 160 kB/s | 81 kB 00:00 2023-12-06T03:50:36.876 INFO:teuthology.orchestra.run.smithi132.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 566 kB/s | 61 kB 00:00 2023-12-06T03:50:36.901 INFO:teuthology.orchestra.run.smithi132.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 334 kB/s | 64 kB 00:00 2023-12-06T03:50:37.117 INFO:teuthology.orchestra.run.smithi132.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.7 MB/s | 1.2 MB 00:00 2023-12-06T03:50:37.120 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:37.121 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2023-12-06T03:50:37.181 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:50:37.196 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:50:37.196 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:50:37.285 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:50:37.288 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:50:37.404 INFO:teuthology.orchestra.run.smithi152.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 762 kB/s | 1.2 MB 00:01 2023-12-06T03:50:37.407 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:37.407 INFO:teuthology.orchestra.run.smithi152.stdout:Total 677 kB/s | 1.9 MB 00:02 2023-12-06T03:50:37.464 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:50:37.480 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:50:37.480 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:50:37.567 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:50:37.570 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:50:37.775 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:50:38.047 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-06T03:50:38.082 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:50:38.236 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-06T03:50:38.321 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-06T03:50:38.463 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-06T03:50:38.517 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-06T03:50:38.684 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-06T03:50:38.745 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-06T03:50:38.894 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-06T03:50:38.941 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-06T03:50:39.133 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-06T03:50:39.181 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-06T03:50:39.419 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-06T03:50:39.431 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-06T03:50:39.659 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-06T03:50:39.705 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-06T03:50:39.947 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-06T03:50:40.680 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-06T03:50:40.959 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-06T03:50:41.010 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-06T03:50:41.159 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-06T03:50:41.266 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-06T03:50:41.368 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-06T03:50:41.491 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-06T03:50:41.511 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-06T03:50:41.708 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-06T03:50:41.818 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-06T03:50:41.818 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-12-06T03:50:41.818 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-06T03:50:41.818 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-06T03:50:41.818 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-06T03:50:41.818 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-06T03:50:41.818 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-06T03:50:41.818 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-06T03:50:41.818 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-06T03:50:41.818 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-06T03:50:41.818 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-06T03:50:41.818 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-06T03:50:41.818 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-06T03:50:41.876 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-06T03:50:42.168 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-12-06T03:50:42.169 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-12-06T03:50:42.169 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-06T03:50:42.169 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-06T03:50:42.169 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-06T03:50:42.169 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-06T03:50:42.169 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-06T03:50:42.169 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-06T03:50:42.169 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-06T03:50:42.169 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-06T03:50:42.169 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-06T03:50:42.169 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-06T03:50:42.169 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-06T03:50:42.578 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-06T03:50:42.579 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:42.579 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:50:42.579 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-12-06T03:50:42.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-06T03:50:42.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-06T03:50:42.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-06T03:50:42.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-06T03:50:42.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-06T03:50:42.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-06T03:50:42.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-06T03:50:42.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-06T03:50:42.579 INFO:teuthology.orchestra.run.smithi132.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-06T03:50:42.580 INFO:teuthology.orchestra.run.smithi132.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-06T03:50:42.580 INFO:teuthology.orchestra.run.smithi132.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-06T03:50:42.580 INFO:teuthology.orchestra.run.smithi132.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-06T03:50:42.580 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:42.580 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:50:42.696 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-mgr-cephadm 2023-12-06T03:50:42.928 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-06T03:50:42.928 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:42.928 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:50:42.928 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-12-06T03:50:42.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-06T03:50:42.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-06T03:50:42.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-06T03:50:42.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-06T03:50:42.928 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-06T03:50:42.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-06T03:50:42.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-06T03:50:42.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-06T03:50:42.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-06T03:50:42.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-06T03:50:42.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-06T03:50:42.929 INFO:teuthology.orchestra.run.smithi152.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-06T03:50:42.929 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:42.929 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:50:43.056 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-mgr-cephadm 2023-12-06T03:50:43.552 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:01:54 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:50:43.910 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:01:56 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:50:45.238 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:50:45.239 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:45.239 INFO:teuthology.orchestra.run.smithi132.stdout: Package Arch Version Repository Size 2023-12-06T03:50:45.239 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:45.239 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:50:45.239 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-12-06T03:50:45.239 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T03:50:45.239 INFO:teuthology.orchestra.run.smithi132.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-06T03:50:45.239 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-06T03:50:45.239 INFO:teuthology.orchestra.run.smithi132.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-06T03:50:45.239 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-06T03:50:45.240 INFO:teuthology.orchestra.run.smithi132.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-06T03:50:45.240 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:45.240 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:50:45.240 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:45.240 INFO:teuthology.orchestra.run.smithi132.stdout:Install 6 Packages 2023-12-06T03:50:45.240 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:45.240 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 5.8 M 2023-12-06T03:50:45.240 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 24 M 2023-12-06T03:50:45.241 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:50:45.592 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:50:45.593 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:45.594 INFO:teuthology.orchestra.run.smithi152.stdout: Package Arch Version Repository Size 2023-12-06T03:50:45.594 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:45.594 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:50:45.594 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-12-06T03:50:45.594 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2023-12-06T03:50:45.594 INFO:teuthology.orchestra.run.smithi152.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-06T03:50:45.594 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-06T03:50:45.594 INFO:teuthology.orchestra.run.smithi152.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-06T03:50:45.594 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-06T03:50:45.594 INFO:teuthology.orchestra.run.smithi152.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-06T03:50:45.594 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:45.594 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:50:45.594 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:45.594 INFO:teuthology.orchestra.run.smithi152.stdout:Install 6 Packages 2023-12-06T03:50:45.595 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:45.595 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 5.8 M 2023-12-06T03:50:45.595 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 24 M 2023-12-06T03:50:45.595 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:50:45.617 INFO:teuthology.orchestra.run.smithi132.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.0 MB/s | 89 kB 00:00 2023-12-06T03:50:45.692 INFO:teuthology.orchestra.run.smithi132.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 4.4 MB/s | 538 kB 00:00 2023-12-06T03:50:45.744 INFO:teuthology.orchestra.run.smithi152.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 89 kB 00:00 2023-12-06T03:50:45.776 INFO:teuthology.orchestra.run.smithi132.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 24 MB/s | 4.8 MB 00:00 2023-12-06T03:50:45.794 INFO:teuthology.orchestra.run.smithi152.stdout:(2/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 5.3 MB/s | 538 kB 00:00 2023-12-06T03:50:45.809 INFO:teuthology.orchestra.run.smithi132.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 110 kB/s | 21 kB 00:00 2023-12-06T03:50:45.886 INFO:teuthology.orchestra.run.smithi152.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 25 MB/s | 4.8 MB 00:00 2023-12-06T03:50:45.993 INFO:teuthology.orchestra.run.smithi132.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 226 kB/s | 49 kB 00:00 2023-12-06T03:50:46.027 INFO:teuthology.orchestra.run.smithi132.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.0 MB/s | 355 kB 00:00 2023-12-06T03:50:46.027 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:46.027 INFO:teuthology.orchestra.run.smithi132.stdout:Total 7.4 MB/s | 5.8 MB 00:00 2023-12-06T03:50:46.080 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:50:46.090 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:50:46.090 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:50:46.190 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:50:46.194 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:50:46.253 INFO:teuthology.orchestra.run.smithi152.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 42 kB/s | 21 kB 00:00 2023-12-06T03:50:46.411 INFO:teuthology.orchestra.run.smithi152.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 95 kB/s | 49 kB 00:00 2023-12-06T03:50:46.471 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:50:46.520 INFO:teuthology.orchestra.run.smithi152.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 490 kB/s | 355 kB 00:00 2023-12-06T03:50:46.522 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:46.523 INFO:teuthology.orchestra.run.smithi152.stdout:Total 6.3 MB/s | 5.8 MB 00:00 2023-12-06T03:50:46.576 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:50:46.585 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:50:46.586 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:50:46.693 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:50:46.697 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:50:46.777 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-06T03:50:46.972 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:50:47.082 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-06T03:50:47.233 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-06T03:50:47.541 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-06T03:50:47.997 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-06T03:50:48.347 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-06T03:50:48.488 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-06T03:50:48.573 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-06T03:50:48.803 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-06T03:50:48.877 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-06T03:50:49.075 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-06T03:50:49.075 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-12-06T03:50:49.075 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-06T03:50:49.076 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-06T03:50:49.076 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-06T03:50:49.076 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-06T03:50:49.088 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-06T03:50:49.308 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-06T03:50:49.771 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-12-06T03:50:49.771 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-12-06T03:50:49.771 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-06T03:50:49.772 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-06T03:50:49.772 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-06T03:50:49.772 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-06T03:50:49.780 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-06T03:50:49.780 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:49.780 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:50:49.780 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-12-06T03:50:49.781 INFO:teuthology.orchestra.run.smithi132.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-12-06T03:50:49.781 INFO:teuthology.orchestra.run.smithi132.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-12-06T03:50:49.781 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:49.781 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:50:49.918 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install ceph-fuse 2023-12-06T03:50:50.463 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-06T03:50:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:50.464 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:50:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-12-06T03:50:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-12-06T03:50:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-12-06T03:50:50.464 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:50.464 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:50:50.608 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install ceph-fuse 2023-12-06T03:50:50.779 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:01 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:50:51.464 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:04 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:50:52.465 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:50:52.465 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:52.465 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2023-12-06T03:50:52.466 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:52.466 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:50:52.466 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-12-06T03:50:52.466 INFO:teuthology.orchestra.run.smithi132.stdout:Installing dependencies: 2023-12-06T03:50:52.466 INFO:teuthology.orchestra.run.smithi132.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-06T03:50:52.466 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:52.466 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:50:52.466 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:52.466 INFO:teuthology.orchestra.run.smithi132.stdout:Install 2 Packages 2023-12-06T03:50:52.466 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:52.467 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 968 k 2023-12-06T03:50:52.467 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 3.2 M 2023-12-06T03:50:52.467 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:50:52.572 INFO:teuthology.orchestra.run.smithi132.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 14 MB/s | 885 kB 00:00 2023-12-06T03:50:52.597 INFO:teuthology.orchestra.run.smithi132.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 937 kB/s | 83 kB 00:00 2023-12-06T03:50:52.597 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:52.597 INFO:teuthology.orchestra.run.smithi132.stdout:Total 7.2 MB/s | 968 kB 00:00 2023-12-06T03:50:52.608 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:50:52.628 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:50:52.629 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:50:52.713 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:50:52.716 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:50:52.883 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:50:53.167 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:50:53.168 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-06T03:50:53.168 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2023-12-06T03:50:53.168 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:53.168 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:50:53.168 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-12-06T03:50:53.169 INFO:teuthology.orchestra.run.smithi152.stdout:Installing dependencies: 2023-12-06T03:50:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-06T03:50:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:53.169 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:50:53.169 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:53.169 INFO:teuthology.orchestra.run.smithi152.stdout:Install 2 Packages 2023-12-06T03:50:53.169 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:53.169 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 968 k 2023-12-06T03:50:53.169 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 3.2 M 2023-12-06T03:50:53.169 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:50:53.256 INFO:teuthology.orchestra.run.smithi152.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 19 MB/s | 885 kB 00:00 2023-12-06T03:50:53.281 INFO:teuthology.orchestra.run.smithi152.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 1.2 MB/s | 83 kB 00:00 2023-12-06T03:50:53.281 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:53.282 INFO:teuthology.orchestra.run.smithi152.stdout:Total 8.4 MB/s | 968 kB 00:00 2023-12-06T03:50:53.292 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:50:53.313 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:50:53.313 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:50:53.367 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-06T03:50:53.398 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:50:53.401 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:50:53.569 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:50:53.754 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-06T03:50:53.754 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-12-06T03:50:53.833 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-06T03:50:54.057 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-06T03:50:54.433 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-12-06T03:50:54.433 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-12-06T03:50:54.447 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-06T03:50:54.447 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:54.447 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:50:54.447 INFO:teuthology.orchestra.run.smithi132.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-06T03:50:54.447 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:54.448 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:50:54.687 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados-devel 2023-12-06T03:50:55.119 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-06T03:50:55.119 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:55.120 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:50:55.120 INFO:teuthology.orchestra.run.smithi152.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-06T03:50:55.120 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:55.120 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:50:55.522 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados-devel 2023-12-06T03:50:55.549 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:06 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:50:56.393 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:09 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:50:57.228 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:50:57.228 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:57.229 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2023-12-06T03:50:57.229 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:57.229 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:50:57.229 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-12-06T03:50:57.229 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:57.229 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:50:57.229 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:50:57.229 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T03:50:57.229 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:57.230 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 131 k 2023-12-06T03:50:57.230 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 439 k 2023-12-06T03:50:57.230 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:50:57.266 INFO:teuthology.orchestra.run.smithi132.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.5 MB/s | 131 kB 00:00 2023-12-06T03:50:57.267 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:57.267 INFO:teuthology.orchestra.run.smithi132.stdout:Total 3.4 MB/s | 131 kB 00:00 2023-12-06T03:50:57.267 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:50:57.272 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:50:57.273 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:50:57.324 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:50:57.327 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:50:57.881 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:50:58.065 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:50:58.066 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:58.066 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2023-12-06T03:50:58.066 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:58.066 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:50:58.066 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-12-06T03:50:58.066 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:58.066 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:50:58.066 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:50:58.066 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2023-12-06T03:50:58.066 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:58.067 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 131 k 2023-12-06T03:50:58.067 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 439 k 2023-12-06T03:50:58.067 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:50:58.109 INFO:teuthology.orchestra.run.smithi152.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 131 kB 00:00 2023-12-06T03:50:58.109 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:50:58.110 INFO:teuthology.orchestra.run.smithi152.stdout:Total 3.0 MB/s | 131 kB 00:00 2023-12-06T03:50:58.110 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:50:58.115 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:50:58.115 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:50:58.165 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:50:58.168 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:50:58.277 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:50:58.466 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:50:58.489 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:50:58.770 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:50:58.801 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:50:59.439 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:50:59.439 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:59.439 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:50:59.439 INFO:teuthology.orchestra.run.smithi132.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-12-06T03:50:59.439 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:50:59.439 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:50:59.511 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:50:59.511 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:59.512 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:50:59.512 INFO:teuthology.orchestra.run.smithi152.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-12-06T03:50:59.512 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:50:59.512 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:50:59.657 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs2 2023-12-06T03:50:59.722 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs2 2023-12-06T03:51:00.515 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:11 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:51:00.586 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:13 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:51:01.928 INFO:teuthology.orchestra.run.smithi132.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:51:02.048 INFO:teuthology.orchestra.run.smithi152.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:51:02.149 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:51:02.151 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T03:51:02.151 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:51:02.255 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install libcephfs-devel 2023-12-06T03:51:02.270 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:51:02.271 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2023-12-06T03:51:02.272 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:51:02.389 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install libcephfs-devel 2023-12-06T03:51:03.114 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:14 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:51:03.246 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:16 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:51:04.816 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:51:04.816 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:51:04.816 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2023-12-06T03:51:04.816 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:51:04.816 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:51:04.816 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-12-06T03:51:04.817 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:04.817 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:51:04.817 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:51:04.817 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T03:51:04.817 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:04.817 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 24 k 2023-12-06T03:51:04.817 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 94 k 2023-12-06T03:51:04.818 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:51:04.857 INFO:teuthology.orchestra.run.smithi132.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 623 kB/s | 24 kB 00:00 2023-12-06T03:51:04.857 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:51:04.857 INFO:teuthology.orchestra.run.smithi132.stdout:Total 603 kB/s | 24 kB 00:00 2023-12-06T03:51:04.857 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:51:04.861 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:51:04.861 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:51:04.863 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:51:04.866 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:51:04.918 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:51:04.977 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:51:04.977 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:51:04.977 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2023-12-06T03:51:04.977 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:51:04.978 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:51:04.978 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-12-06T03:51:04.978 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:04.978 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:51:04.978 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:51:04.978 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2023-12-06T03:51:04.978 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:04.978 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 24 k 2023-12-06T03:51:04.979 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 94 k 2023-12-06T03:51:04.979 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:51:05.015 INFO:teuthology.orchestra.run.smithi152.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 678 kB/s | 24 kB 00:00 2023-12-06T03:51:05.015 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:51:05.015 INFO:teuthology.orchestra.run.smithi152.stdout:Total 656 kB/s | 24 kB 00:00 2023-12-06T03:51:05.015 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:51:05.019 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:51:05.019 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:51:05.021 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:51:05.024 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:51:05.076 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:51:05.122 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:05.254 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:05.413 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:05.520 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:06.063 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:06.063 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:06.064 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:51:06.064 INFO:teuthology.orchestra.run.smithi132.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-12-06T03:51:06.064 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:06.064 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:51:06.169 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:06.170 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:06.170 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:51:06.170 INFO:teuthology.orchestra.run.smithi152.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-12-06T03:51:06.170 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:06.170 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:51:06.188 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librados2 2023-12-06T03:51:06.325 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librados2 2023-12-06T03:51:07.041 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:18 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:51:07.184 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:20 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:51:08.517 INFO:teuthology.orchestra.run.smithi132.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:51:08.640 INFO:teuthology.orchestra.run.smithi152.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:51:08.742 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:51:08.742 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T03:51:08.742 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:51:08.849 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install librbd1 2023-12-06T03:51:08.864 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:51:08.865 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2023-12-06T03:51:08.865 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:51:08.972 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install librbd1 2023-12-06T03:51:09.700 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:20 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:51:09.826 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:22 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:51:11.150 INFO:teuthology.orchestra.run.smithi132.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:51:11.315 INFO:teuthology.orchestra.run.smithi152.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:51:11.373 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:51:11.374 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T03:51:11.374 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:51:11.482 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rados 2023-12-06T03:51:11.540 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:51:11.541 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2023-12-06T03:51:11.541 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:51:11.649 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rados 2023-12-06T03:51:12.339 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:23 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:51:12.506 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:25 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:51:13.782 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:51:13.953 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:51:14.008 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:51:14.010 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T03:51:14.010 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:51:14.120 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rgw 2023-12-06T03:51:14.178 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:51:14.179 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2023-12-06T03:51:14.179 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:51:14.275 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rgw 2023-12-06T03:51:14.983 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:25 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:51:15.129 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:28 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:51:16.430 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:51:16.583 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:51:16.654 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:51:16.655 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T03:51:16.656 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:51:16.764 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-cephfs 2023-12-06T03:51:16.808 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:51:16.809 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2023-12-06T03:51:16.809 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:51:16.900 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-cephfs 2023-12-06T03:51:17.621 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:28 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:51:17.752 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:30 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:51:19.056 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:51:19.195 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:51:19.281 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:51:19.282 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T03:51:19.282 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:51:19.399 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install python3-rbd 2023-12-06T03:51:19.422 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:51:19.423 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2023-12-06T03:51:19.423 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:51:19.529 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install python3-rbd 2023-12-06T03:51:20.259 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:31 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:51:20.383 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:33 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:51:21.713 INFO:teuthology.orchestra.run.smithi132.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:51:21.819 INFO:teuthology.orchestra.run.smithi152.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-12-06T03:51:21.939 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:51:21.940 INFO:teuthology.orchestra.run.smithi132.stdout:Nothing to do. 2023-12-06T03:51:21.940 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:51:22.046 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:51:22.047 INFO:teuthology.orchestra.run.smithi152.stdout:Nothing to do. 2023-12-06T03:51:22.047 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:51:22.067 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-fuse 2023-12-06T03:51:22.165 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-fuse 2023-12-06T03:51:22.918 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:33 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:51:23.017 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:36 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:51:24.616 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:51:24.616 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:51:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2023-12-06T03:51:24.617 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:51:24.617 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:51:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-12-06T03:51:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:24.617 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:51:24.617 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:51:24.617 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T03:51:24.617 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:24.618 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 89 k 2023-12-06T03:51:24.618 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 298 k 2023-12-06T03:51:24.618 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:51:24.663 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.9 MB/s | 89 kB 00:00 2023-12-06T03:51:24.663 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:51:24.664 INFO:teuthology.orchestra.run.smithi132.stdout:Total 1.9 MB/s | 89 kB 00:00 2023-12-06T03:51:24.664 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:51:24.673 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:51:24.674 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:51:24.700 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:51:24.700 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:51:24.700 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2023-12-06T03:51:24.700 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:51:24.700 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:51:24.700 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-12-06T03:51:24.700 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:24.700 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:51:24.700 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:51:24.701 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2023-12-06T03:51:24.701 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:24.701 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 89 k 2023-12-06T03:51:24.701 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 298 k 2023-12-06T03:51:24.701 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:51:24.724 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:51:24.727 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:51:24.745 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 2.0 MB/s | 89 kB 00:00 2023-12-06T03:51:24.746 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:51:24.746 INFO:teuthology.orchestra.run.smithi152.stdout:Total 1.9 MB/s | 89 kB 00:00 2023-12-06T03:51:24.746 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:51:24.756 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:51:24.756 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:51:24.807 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:51:24.810 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:51:24.843 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:51:24.917 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:51:25.028 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:25.110 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:25.317 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:25.382 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:25.982 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:25.982 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:25.982 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:51:25.982 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-12-06T03:51:25.982 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:25.982 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:51:26.035 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:26.035 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:26.035 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:51:26.035 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-12-06T03:51:26.035 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:26.035 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:51:26.215 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-mirror 2023-12-06T03:51:26.221 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-mirror 2023-12-06T03:51:27.078 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:40 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:51:27.082 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:38 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:51:28.762 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:51:28.763 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:51:28.763 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2023-12-06T03:51:28.763 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:51:28.763 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:51:28.763 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-06T03:51:28.763 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:28.763 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:51:28.763 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:51:28.763 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T03:51:28.763 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:28.764 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 3.7 M 2023-12-06T03:51:28.764 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 18 M 2023-12-06T03:51:28.764 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:51:28.776 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:51:28.777 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:51:28.777 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2023-12-06T03:51:28.777 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:51:28.777 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:51:28.777 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-12-06T03:51:28.777 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:28.777 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:51:28.777 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:51:28.777 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2023-12-06T03:51:28.777 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:28.778 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 3.7 M 2023-12-06T03:51:28.778 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 18 M 2023-12-06T03:51:28.778 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:51:28.849 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 43 MB/s | 3.7 MB 00:00 2023-12-06T03:51:28.850 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:51:28.850 INFO:teuthology.orchestra.run.smithi132.stdout:Total 43 MB/s | 3.7 MB 00:00 2023-12-06T03:51:28.850 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:51:28.862 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:51:28.862 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:51:28.881 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 36 MB/s | 3.7 MB 00:00 2023-12-06T03:51:28.881 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:51:28.882 INFO:teuthology.orchestra.run.smithi152.stdout:Total 36 MB/s | 3.7 MB 00:00 2023-12-06T03:51:28.882 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:51:28.894 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:51:28.894 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:51:28.924 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:51:28.927 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:51:28.957 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:51:28.960 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:51:29.392 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:51:29.428 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:51:29.587 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:29.627 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:30.016 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:30.048 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:30.724 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:30.724 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:30.724 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:51:30.724 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-12-06T03:51:30.724 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:30.724 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:51:30.744 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:30.744 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:30.744 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:51:30.744 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-12-06T03:51:30.744 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:30.744 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:51:30.925 DEBUG:teuthology.orchestra.run.smithi152:> sudo yum -y install rbd-nbd 2023-12-06T03:51:30.932 DEBUG:teuthology.orchestra.run.smithi132:> sudo yum -y install rbd-nbd 2023-12-06T03:51:31.790 INFO:teuthology.orchestra.run.smithi132.stdout:Last metadata expiration check: 0:02:42 ago on Wed 06 Dec 2023 03:48:49 AM UTC. 2023-12-06T03:51:31.794 INFO:teuthology.orchestra.run.smithi152.stdout:Last metadata expiration check: 0:02:44 ago on Wed 06 Dec 2023 03:48:47 AM UTC. 2023-12-06T03:51:33.455 INFO:teuthology.orchestra.run.smithi132.stdout:Dependencies resolved. 2023-12-06T03:51:33.456 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:51:33.456 INFO:teuthology.orchestra.run.smithi132.stdout: Package Architecture Version Repository Size 2023-12-06T03:51:33.456 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:51:33.456 INFO:teuthology.orchestra.run.smithi132.stdout:Installing: 2023-12-06T03:51:33.456 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-12-06T03:51:33.456 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:33.456 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction Summary 2023-12-06T03:51:33.456 INFO:teuthology.orchestra.run.smithi132.stdout:================================================================================ 2023-12-06T03:51:33.456 INFO:teuthology.orchestra.run.smithi132.stdout:Install 1 Package 2023-12-06T03:51:33.456 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:33.456 INFO:teuthology.orchestra.run.smithi132.stdout:Total download size: 173 k 2023-12-06T03:51:33.457 INFO:teuthology.orchestra.run.smithi132.stdout:Installed size: 649 k 2023-12-06T03:51:33.457 INFO:teuthology.orchestra.run.smithi132.stdout:Downloading Packages: 2023-12-06T03:51:33.482 INFO:teuthology.orchestra.run.smithi152.stdout:Dependencies resolved. 2023-12-06T03:51:33.483 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:51:33.483 INFO:teuthology.orchestra.run.smithi152.stdout: Package Architecture Version Repository Size 2023-12-06T03:51:33.483 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:51:33.483 INFO:teuthology.orchestra.run.smithi152.stdout:Installing: 2023-12-06T03:51:33.483 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-12-06T03:51:33.483 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:33.483 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction Summary 2023-12-06T03:51:33.483 INFO:teuthology.orchestra.run.smithi152.stdout:================================================================================ 2023-12-06T03:51:33.483 INFO:teuthology.orchestra.run.smithi152.stdout:Install 1 Package 2023-12-06T03:51:33.483 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:33.484 INFO:teuthology.orchestra.run.smithi152.stdout:Total download size: 173 k 2023-12-06T03:51:33.484 INFO:teuthology.orchestra.run.smithi152.stdout:Installed size: 649 k 2023-12-06T03:51:33.484 INFO:teuthology.orchestra.run.smithi152.stdout:Downloading Packages: 2023-12-06T03:51:33.494 INFO:teuthology.orchestra.run.smithi132.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 173 kB 00:00 2023-12-06T03:51:33.495 INFO:teuthology.orchestra.run.smithi132.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:51:33.495 INFO:teuthology.orchestra.run.smithi132.stdout:Total 4.4 MB/s | 173 kB 00:00 2023-12-06T03:51:33.495 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction check 2023-12-06T03:51:33.505 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction check succeeded. 2023-12-06T03:51:33.506 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction test 2023-12-06T03:51:33.528 INFO:teuthology.orchestra.run.smithi152.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.8 MB/s | 173 kB 00:00 2023-12-06T03:51:33.529 INFO:teuthology.orchestra.run.smithi152.stdout:-------------------------------------------------------------------------------- 2023-12-06T03:51:33.529 INFO:teuthology.orchestra.run.smithi152.stdout:Total 3.7 MB/s | 173 kB 00:00 2023-12-06T03:51:33.529 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction check 2023-12-06T03:51:33.539 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction check succeeded. 2023-12-06T03:51:33.540 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction test 2023-12-06T03:51:33.555 INFO:teuthology.orchestra.run.smithi132.stdout:Transaction test succeeded. 2023-12-06T03:51:33.558 INFO:teuthology.orchestra.run.smithi132.stdout:Running transaction 2023-12-06T03:51:33.591 INFO:teuthology.orchestra.run.smithi152.stdout:Transaction test succeeded. 2023-12-06T03:51:33.594 INFO:teuthology.orchestra.run.smithi152.stdout:Running transaction 2023-12-06T03:51:33.674 INFO:teuthology.orchestra.run.smithi132.stdout: Preparing : 1/1 2023-12-06T03:51:33.711 INFO:teuthology.orchestra.run.smithi152.stdout: Preparing : 1/1 2023-12-06T03:51:33.903 INFO:teuthology.orchestra.run.smithi152.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:33.919 INFO:teuthology.orchestra.run.smithi132.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:34.190 INFO:teuthology.orchestra.run.smithi152.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:34.210 INFO:teuthology.orchestra.run.smithi132.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:34.874 INFO:teuthology.orchestra.run.smithi132.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:34.874 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:34.875 INFO:teuthology.orchestra.run.smithi132.stdout:Installed: 2023-12-06T03:51:34.875 INFO:teuthology.orchestra.run.smithi132.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-12-06T03:51:34.875 INFO:teuthology.orchestra.run.smithi132.stdout: 2023-12-06T03:51:34.875 INFO:teuthology.orchestra.run.smithi132.stdout:Complete! 2023-12-06T03:51:34.896 INFO:teuthology.orchestra.run.smithi152.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-12-06T03:51:34.896 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:34.896 INFO:teuthology.orchestra.run.smithi152.stdout:Installed: 2023-12-06T03:51:34.896 INFO:teuthology.orchestra.run.smithi152.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-12-06T03:51:34.896 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:51:34.896 INFO:teuthology.orchestra.run.smithi152.stdout:Complete! 2023-12-06T03:51:35.085 DEBUG:teuthology.parallel:result is None 2023-12-06T03:51:35.116 DEBUG:teuthology.parallel:result is None 2023-12-06T03:51:35.116 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-06T03:51:35.117 INFO:teuthology.packaging:ref: None 2023-12-06T03:51:35.117 INFO:teuthology.packaging:tag: v16.2.4 2023-12-06T03:51:35.117 INFO:teuthology.packaging:branch: None 2023-12-06T03:51:35.117 INFO:teuthology.packaging:sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b 2023-12-06T03:51:35.300 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-06T03:51:35.301 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-06T03:51:35.451 DEBUG:teuthology.orchestra.run.smithi132:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-06T03:51:35.474 INFO:teuthology.orchestra.run.smithi132.stdout:16.2.4-0.el8 2023-12-06T03:51:35.474 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-12-06T03:51:35.475 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-12-06T03:51:35.475 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-12-06T03:51:35.475 INFO:teuthology.packaging:ref: None 2023-12-06T03:51:35.475 INFO:teuthology.packaging:tag: v16.2.4 2023-12-06T03:51:35.475 INFO:teuthology.packaging:branch: None 2023-12-06T03:51:35.475 INFO:teuthology.packaging:sha1: d03fca4f35d43f61b4679936aef7bd2f01cced0b 2023-12-06T03:51:35.673 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-06T03:51:35.673 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-12-06T03:51:35.899 DEBUG:teuthology.orchestra.run.smithi152:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-06T03:51:35.923 INFO:teuthology.orchestra.run.smithi152.stdout:16.2.4-0.el8 2023-12-06T03:51:35.924 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-12-06T03:51:35.924 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-12-06T03:51:35.924 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-06T03:51:35.924 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T03:51:35.924 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-06T03:51:35.954 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-06T03:51:35.955 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-06T03:51:35.993 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-06T03:51:35.993 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T03:51:35.993 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/daemon-helper 2023-12-06T03:51:36.023 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-06T03:51:36.091 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-06T03:51:36.092 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/daemon-helper 2023-12-06T03:51:36.120 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-06T03:51:36.188 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-06T03:51:36.188 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T03:51:36.188 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-06T03:51:36.216 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-06T03:51:36.284 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-06T03:51:36.285 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-06T03:51:36.312 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-06T03:51:36.379 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-06T03:51:36.379 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T03:51:36.379 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd of=/usr/bin/stdin-killer 2023-12-06T03:51:36.405 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-06T03:51:36.472 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-06T03:51:36.472 DEBUG:teuthology.orchestra.run.smithi152:> sudo dd of=/usr/bin/stdin-killer 2023-12-06T03:51:36.499 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-06T03:51:36.566 INFO:teuthology.run_tasks:Running task print... 2023-12-06T03:51:36.576 INFO:teuthology.task.print:**** done install task... 2023-12-06T03:51:36.577 INFO:teuthology.run_tasks:Running task cephadm... 2023-12-06T03:51:36.690 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'd03fca4f35d43f61b4679936aef7bd2f01cced0b'} 2023-12-06T03:51:36.690 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-12-06T03:51:36.691 INFO:tasks.cephadm:Cluster fsid is c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:51:36.691 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-12-06T03:51:36.691 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-12-06T03:51:36.691 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi132': '172.21.15.132', 'mon.smithi152': '172.21.15.152'} 2023-12-06T03:51:36.691 INFO:tasks.cephadm:Normalizing hostnames... 2023-12-06T03:51:36.691 DEBUG:teuthology.orchestra.run.smithi132:> sudo hostname $(hostname -s) 2023-12-06T03:51:36.719 DEBUG:teuthology.orchestra.run.smithi152:> sudo hostname $(hostname -s) 2023-12-06T03:51:36.747 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-12-06T03:51:36.747 DEBUG:teuthology.orchestra.run.smithi132:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-06T03:51:36.990 INFO:teuthology.orchestra.run.smithi132.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Dec 6 03:51 /home/ubuntu/cephtest/cephadm 2023-12-06T03:51:36.991 DEBUG:teuthology.orchestra.run.smithi152:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-06T03:51:37.135 INFO:teuthology.orchestra.run.smithi152.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Dec 6 03:51 /home/ubuntu/cephtest/cephadm 2023-12-06T03:51:37.136 DEBUG:teuthology.orchestra.run.smithi132:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-06T03:51:37.155 DEBUG:teuthology.orchestra.run.smithi152:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-06T03:51:37.184 DEBUG:teuthology.orchestra.run.smithi132:> sudo mkdir -p /etc/ceph 2023-12-06T03:51:37.223 DEBUG:teuthology.orchestra.run.smithi152:> sudo mkdir -p /etc/ceph 2023-12-06T03:51:37.250 DEBUG:teuthology.orchestra.run.smithi132:> sudo chmod 777 /etc/ceph 2023-12-06T03:51:37.290 DEBUG:teuthology.orchestra.run.smithi152:> sudo chmod 777 /etc/ceph 2023-12-06T03:51:37.317 INFO:tasks.cephadm:Writing seed config... 2023-12-06T03:51:37.318 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-12-06T03:51:37.318 INFO:tasks.cephadm: override: [client] debug client = 20 2023-12-06T03:51:37.318 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-12-06T03:51:37.318 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-12-06T03:51:37.318 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-12-06T03:51:37.318 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-12-06T03:51:37.319 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-12-06T03:51:37.320 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-12-06T03:51:37.321 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T03:51:37.321 DEBUG:teuthology.orchestra.run.smithi132:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-12-06T03:51:37.348 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = c121c6d8-93ea-11ee-95a2-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-12-06T03:51:37.349 DEBUG:teuthology.orchestra.run.smithi132:mon.smithi132> sudo journalctl -f -n 0 -u ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi132.service 2023-12-06T03:51:37.391 INFO:tasks.cephadm:Bootstrapping... 2023-12-06T03:51:37.391 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.132 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-12-06T03:51:37.417 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:-- Logs begin at Wed 2023-12-06 03:39:28 UTC. -- 2023-12-06T03:51:37.654 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-06T03:51:38.545 INFO:teuthology.orchestra.run.smithi132.stderr:/bin/podman: 4.6.1 2023-12-06T03:51:38.556 INFO:teuthology.orchestra.run.smithi132.stderr:Verifying podman|docker is present... 2023-12-06T03:51:38.556 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-06T03:51:38.578 INFO:teuthology.orchestra.run.smithi132.stderr:/bin/podman: 4.6.1 2023-12-06T03:51:38.591 INFO:teuthology.orchestra.run.smithi132.stderr:Verifying lvm2 is present... 2023-12-06T03:51:38.591 INFO:teuthology.orchestra.run.smithi132.stderr:Verifying time synchronization is in place... 2023-12-06T03:51:38.591 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-enabled chrony.service 2023-12-06T03:51:38.599 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-06T03:51:38.599 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-active chrony.service 2023-12-06T03:51:38.605 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: inactive 2023-12-06T03:51:38.606 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-enabled chronyd.service 2023-12-06T03:51:38.613 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: enabled 2023-12-06T03:51:38.613 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-active chronyd.service 2023-12-06T03:51:38.620 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: active 2023-12-06T03:51:38.620 INFO:teuthology.orchestra.run.smithi132.stderr:Unit chronyd.service is enabled and running 2023-12-06T03:51:38.620 INFO:teuthology.orchestra.run.smithi132.stderr:Repeating the final host check... 2023-12-06T03:51:38.620 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-12-06T03:51:38.640 INFO:teuthology.orchestra.run.smithi132.stderr:/bin/podman: 4.6.1 2023-12-06T03:51:38.656 INFO:teuthology.orchestra.run.smithi132.stderr:podman|docker (/bin/podman) is present 2023-12-06T03:51:38.656 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl is present 2023-12-06T03:51:38.656 INFO:teuthology.orchestra.run.smithi132.stderr:lvcreate is present 2023-12-06T03:51:38.656 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-enabled chrony.service 2023-12-06T03:51:38.661 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-06T03:51:38.662 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-active chrony.service 2023-12-06T03:51:38.667 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: inactive 2023-12-06T03:51:38.668 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-enabled chronyd.service 2023-12-06T03:51:38.674 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: enabled 2023-12-06T03:51:38.674 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-active chronyd.service 2023-12-06T03:51:38.679 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: active 2023-12-06T03:51:38.680 INFO:teuthology.orchestra.run.smithi132.stderr:Unit chronyd.service is enabled and running 2023-12-06T03:51:38.680 INFO:teuthology.orchestra.run.smithi132.stderr:Host looks OK 2023-12-06T03:51:38.680 INFO:teuthology.orchestra.run.smithi132.stderr:Cluster fsid: c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:51:38.680 INFO:teuthology.orchestra.run.smithi132.stderr:Acquiring lock 139868735597032 on /run/cephadm/c121c6d8-93ea-11ee-95a2-87774f69a715.lock 2023-12-06T03:51:38.681 INFO:teuthology.orchestra.run.smithi132.stderr:Lock 139868735597032 acquired on /run/cephadm/c121c6d8-93ea-11ee-95a2-87774f69a715.lock 2023-12-06T03:51:38.681 INFO:teuthology.orchestra.run.smithi132.stderr:Verifying IP 172.21.15.132 port 3300 ... 2023-12-06T03:51:38.681 INFO:teuthology.orchestra.run.smithi132.stderr:Verifying IP 172.21.15.132 port 6789 ... 2023-12-06T03:51:38.681 INFO:teuthology.orchestra.run.smithi132.stderr:Base mon IP is 172.21.15.132, final addrv is [v2:172.21.15.132:3300,v1:172.21.15.132:6789] 2023-12-06T03:51:38.682 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /sbin/ip route ls 2023-12-06T03:51:38.683 INFO:teuthology.orchestra.run.smithi132.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.132 metric 100 2023-12-06T03:51:38.684 INFO:teuthology.orchestra.run.smithi132.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.132 metric 100 2023-12-06T03:51:38.684 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /sbin/ip -6 route ls 2023-12-06T03:51:38.686 INFO:teuthology.orchestra.run.smithi132.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-12-06T03:51:38.687 INFO:teuthology.orchestra.run.smithi132.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2023-12-06T03:51:38.687 INFO:teuthology.orchestra.run.smithi132.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1787sec hoplimit 64 pref medium 2023-12-06T03:51:38.687 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /sbin/ip -6 addr ls 2023-12-06T03:51:38.689 INFO:teuthology.orchestra.run.smithi132.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-12-06T03:51:38.689 INFO:teuthology.orchestra.run.smithi132.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-12-06T03:51:38.690 INFO:teuthology.orchestra.run.smithi132.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-12-06T03:51:38.690 INFO:teuthology.orchestra.run.smithi132.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-12-06T03:51:38.690 INFO:teuthology.orchestra.run.smithi132.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:773f/64 scope link 2023-12-06T03:51:38.690 INFO:teuthology.orchestra.run.smithi132.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-12-06T03:51:38.691 INFO:teuthology.orchestra.run.smithi132.stderr:Mon IP 172.21.15.132 is in CIDR network 172.21.0.0/20 2023-12-06T03:51:38.691 INFO:teuthology.orchestra.run.smithi132.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-12-06T03:51:38.692 INFO:teuthology.orchestra.run.smithi132.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-12-06T03:51:38.692 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-12-06T03:51:38.798 INFO:teuthology.orchestra.run.smithi132.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-12-06T03:51:45.353 INFO:teuthology.orchestra.run.smithi132.stderr:/bin/podman: Getting image source signatures 2023-12-06T03:51:45.366 INFO:teuthology.orchestra.run.smithi132.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-12-06T03:51:45.366 INFO:teuthology.orchestra.run.smithi132.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-12-06T03:51:57.483 INFO:teuthology.orchestra.run.smithi132.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-12-06T03:51:57.565 INFO:teuthology.orchestra.run.smithi132.stderr:/bin/podman: Writing manifest to image destination 2023-12-06T03:51:57.869 INFO:teuthology.orchestra.run.smithi132.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-12-06T03:51:57.886 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-12-06T03:51:58.402 INFO:teuthology.orchestra.run.smithi132.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-12-06T03:51:58.750 INFO:teuthology.orchestra.run.smithi132.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-12-06T03:51:58.750 INFO:teuthology.orchestra.run.smithi132.stderr:Extracting ceph user uid/gid from container image... 2023-12-06T03:51:58.750 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-12-06T03:51:59.039 INFO:teuthology.orchestra.run.smithi132.stderr:stat: 167 167 2023-12-06T03:51:59.372 INFO:teuthology.orchestra.run.smithi132.stderr:Creating initial keys... 2023-12-06T03:51:59.373 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-06T03:51:59.649 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-authtool: AQDf729l2ERXJhAAraEZgWpi6pKxk72qHD+p1g== 2023-12-06T03:52:00.020 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-06T03:52:00.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-authtool: AQDg729lfUXzEBAAq3RJ2oTP4ii9ASon8y4uXQ== 2023-12-06T03:52:00.635 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-12-06T03:52:00.899 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-authtool: AQDg729lFTgNNRAA9tr9ODnEfwTYI6OgV83Gpw== 2023-12-06T03:52:01.247 INFO:teuthology.orchestra.run.smithi132.stderr:Creating initial monmap... 2023-12-06T03:52:01.247 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmp5mtj87a4:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 --addv smithi132 [v2:172.21.15.132:3300,v1:172.21.15.132:6789] /tmp/monmap 2023-12-06T03:52:01.533 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-06T03:52:01.533 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:52:01.533 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-06T03:52:01.868 INFO:teuthology.orchestra.run.smithi132.stderr:monmaptool for smithi132 [v2:172.21.15.132:3300,v1:172.21.15.132:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-06T03:52:01.868 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/monmaptool: set fsid to c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:52:01.868 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-06T03:52:01.869 INFO:teuthology.orchestra.run.smithi132.stderr: 2023-12-06T03:52:01.869 INFO:teuthology.orchestra.run.smithi132.stderr:Creating mon... 2023-12-06T03:52:01.869 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/mon.smithi132:/var/lib/ceph/mon/ceph-smithi132:z -v /tmp/ceph-tmptwiuwig3:/tmp/keyring:z -v /tmp/ceph-tmp5mtj87a4:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi132 --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-12-06T03:52:02.170 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.142+0000 7f39b11bf700 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-06T03:52:02.170 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.142+0000 7f39b11bf700 1 imported monmap: 2023-12-06T03:52:02.171 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: epoch 0 2023-12-06T03:52:02.171 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: fsid c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:52:02.171 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: last_changed 2023-12-06T03:52:01.531218+0000 2023-12-06T03:52:02.171 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: created 2023-12-06T03:52:01.531218+0000 2023-12-06T03:52:02.171 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-12-06T03:52:02.171 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-12-06T03:52:02.171 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon.smithi132 2023-12-06T03:52:02.171 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.171 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.142+0000 7f39b11bf700 0 /usr/bin/ceph-mon: set fsid to c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:52:02.171 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: RocksDB version: 6.8.1 2023-12-06T03:52:02.171 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.171 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-06T03:52:02.171 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Compile date May 13 2021 2023-12-06T03:52:02.171 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: DB SUMMARY 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi132/store.db dir, Total Num: 0, files: 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi132/store.db: 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.error_if_exists: 0 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.create_if_missing: 1 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.paranoid_checks: 1 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.env: 0x5562958291c0 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.fs: Posix File System 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.info_log: 0x5562980a0ec0 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.statistics: (nil) 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.use_fsync: 0 2023-12-06T03:52:02.172 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.max_log_file_size: 0 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.allow_fallocate: 1 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.use_direct_reads: 0 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.create_missing_column_families: 0 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.db_log_dir: 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi132/store.db 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.max_subcompactions: 1 2023-12-06T03:52:02.173 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.max_background_flushes: -1 2023-12-06T03:52:02.174 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-06T03:52:02.174 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-06T03:52:02.174 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-06T03:52:02.174 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-06T03:52:02.174 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-06T03:52:02.174 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.advise_random_on_open: 1 2023-12-06T03:52:02.174 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-06T03:52:02.174 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.write_buffer_manager: 0x5562980a7530 2023-12-06T03:52:02.174 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-06T03:52:02.174 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-06T03:52:02.174 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-06T03:52:02.174 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-06T03:52:02.174 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.rate_limiter: (nil) 2023-12-06T03:52:02.174 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.unordered_write: 0 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.row_cache: None 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.wal_filter: None 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.preserve_deletes: 0 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.two_write_queues: 0 2023-12-06T03:52:02.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.manual_wal_flush: 0 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.atomic_flush: 0 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.log_readahead_size: 0 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.max_background_jobs: 2 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.max_background_compactions: -1 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.max_total_wal_size: 0 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-06T03:52:02.176 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.max_open_files: -1 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.bytes_per_sync: 0 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Compression algorithms supported: 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: kZSTD supported: 0 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: kXpressCompression supported: 0 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: kLZ4Compression supported: 1 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: kBZip2Compression supported: 0 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: kZlibCompression supported: 1 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: kSnappyCompression supported: 1 2023-12-06T03:52:02.177 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.143+0000 7f39b11bf700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi132/store.db/MANIFEST-000001 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.merge_operator: 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compaction_filter: None 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compaction_filter_factory: None 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556297fb8120) 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-12-06T03:52:02.178 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: index_type: 0 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: checksum: 1 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: block_cache: 0x556297feed10 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-12-06T03:52:02.179 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: format_version: 2 2023-12-06T03:52:02.180 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: block_align: 0 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compression: NoCompression 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.num_levels: 7 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-06T03:52:02.181 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compression_opts.level: 32767 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compression_opts.enabled: false 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-06T03:52:02.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.arena_block_size: 4194304 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-06T03:52:02.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.table_properties_collectors: 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.inplace_update_support: 0 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-06T03:52:02.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.bloom_locality: 0 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.max_successive_merges: 0 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.force_consistency_checks: 0 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.ttl: 2592000 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi132/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.144+0000 7f39b11bf700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.145+0000 7f39b11bf700 4 rocksdb: DB pointer 0x5562980b3800 2023-12-06T03:52:02.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.145+0000 7f399a058700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-12-06T03:52:02.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.145+0000 7f399a058700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-12-06T03:52:02.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-12-06T03:52:02.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-06T03:52:02.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-06T03:52:02.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-06T03:52:02.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-06T03:52:02.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-06T03:52:02.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-06T03:52:02.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-06T03:52:02.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-06T03:52:02.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-06T03:52:02.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-06T03:52:02.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.187 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-06T03:52:02.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-06T03:52:02.189 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-06T03:52:02.189 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-06T03:52:02.189 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-06T03:52:02.189 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-06T03:52:02.189 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.189 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-06T03:52:02.189 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: 2023-12-06T03:52:02.189 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.146+0000 7f39b11bf700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-12-06T03:52:02.189 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.146+0000 7f39b11bf700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-12-06T03:52:02.189 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph-mon: debug 2023-12-06T03:52:02.146+0000 7f39b11bf700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi132 for mon.smithi132 2023-12-06T03:52:02.484 INFO:teuthology.orchestra.run.smithi132.stderr:create mon.smithi132 on 2023-12-06T03:52:02.485 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:52:02.489 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl disable ceph.target 2023-12-06T03:52:02.496 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-12-06T03:52:02.586 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl enable ceph.target 2023-12-06T03:52:02.593 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-12-06T03:52:02.678 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl start ceph.target 2023-12-06T03:52:02.685 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl enable ceph-c121c6d8-93ea-11ee-95a2-87774f69a715.target 2023-12-06T03:52:02.690 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-c121c6d8-93ea-11ee-95a2-87774f69a715.target → /etc/systemd/system/ceph-c121c6d8-93ea-11ee-95a2-87774f69a715.target. 2023-12-06T03:52:02.690 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-c121c6d8-93ea-11ee-95a2-87774f69a715.target → /etc/systemd/system/ceph-c121c6d8-93ea-11ee-95a2-87774f69a715.target. 2023-12-06T03:52:02.768 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl start ceph-c121c6d8-93ea-11ee-95a2-87774f69a715.target 2023-12-06T03:52:02.775 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl daemon-reload 2023-12-06T03:52:02.860 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl stop ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi132 2023-12-06T03:52:02.891 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl reset-failed ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi132 2023-12-06T03:52:02.896 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: Failed to reset failed state of unit ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi132.service: Unit ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi132.service not loaded. 2023-12-06T03:52:02.897 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl enable ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi132 2023-12-06T03:52:02.903 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: Created symlink /etc/systemd/system/ceph-c121c6d8-93ea-11ee-95a2-87774f69a715.target.wants/ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi132.service → /etc/systemd/system/ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@.service. 2023-12-06T03:52:02.991 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl start ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi132 2023-12-06T03:52:03.030 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:02 smithi132 systemd[1]: Starting Ceph mon.smithi132 for c121c6d8-93ea-11ee-95a2-87774f69a715... 2023-12-06T03:52:03.295 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-06T03:52:03.303 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: disabled 2023-12-06T03:52:03.304 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-active firewalld.service 2023-12-06T03:52:03.312 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: inactive 2023-12-06T03:52:03.312 INFO:teuthology.orchestra.run.smithi132.stderr:firewalld.service is not enabled 2023-12-06T03:52:03.312 INFO:teuthology.orchestra.run.smithi132.stderr:Not possible to enable service . firewalld.service is not available 2023-12-06T03:52:03.312 INFO:teuthology.orchestra.run.smithi132.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-12-06T03:52:03.313 INFO:teuthology.orchestra.run.smithi132.stderr:Waiting for mon to start... 2023-12-06T03:52:03.313 INFO:teuthology.orchestra.run.smithi132.stderr:Waiting for mon... 2023-12-06T03:52:03.313 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/mon.smithi132:/var/lib/ceph/mon/ceph-smithi132:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-12-06T03:52:03.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:03 smithi132 bash[111275]: 240c1f38f32cf290415be14c3f033015ee5f2b71c4f87b7ca0a598bdd6ce4515 2023-12-06T03:52:03.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:03 smithi132 systemd[1]: Started Ceph mon.smithi132 for c121c6d8-93ea-11ee-95a2-87774f69a715. 2023-12-06T03:52:03.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111290]: cluster 2023-12-06T03:52:03.326656+0000 mon.smithi132 (mon.0) 0 : [INF] mkfs c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:52:03.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111290]: cluster 2023-12-06T03:52:03.310059+0000 mon.smithi132 (mon.0) 1 : cluster [INF] mon.smithi132 is new leader, mons smithi132 in quorum (ranks 0) 2023-12-06T03:52:03.705 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.701+0000 7f7e5887e700 1 Processor -- start 2023-12-06T03:52:03.705 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.701+0000 7f7e5887e700 1 -- start start 2023-12-06T03:52:03.705 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.702+0000 7f7e5887e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e540fa0a0 0x7f7e540fa4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:03.706 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.702+0000 7f7e5887e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e540fa9d0 con 0x7f7e540fa0a0 2023-12-06T03:52:03.706 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.702+0000 7f7e5259c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e540fa0a0 0x7f7e540fa4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:03.706 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.702+0000 7f7e5259c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e540fa0a0 0x7f7e540fa4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:53050/0 (socket says 172.21.15.132:53050) 2023-12-06T03:52:03.706 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.702+0000 7f7e5259c700 1 -- 172.21.15.132:0/3180665036 learned_addr learned my addr 172.21.15.132:0/3180665036 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:03.706 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.702+0000 7f7e5259c700 1 -- 172.21.15.132:0/3180665036 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e540fab10 con 0x7f7e540fa0a0 2023-12-06T03:52:03.706 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.702+0000 7f7e5259c700 1 --2- 172.21.15.132:0/3180665036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e540fa0a0 0x7f7e540fa4e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7e4400b5f0 tx=0x7f7e44006ee0).ready entity=mon.0 client_cookie=75bb5858665e9417 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:03.706 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.702+0000 7f7e5159a700 1 -- 172.21.15.132:0/3180665036 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e4400d600 con 0x7f7e540fa0a0 2023-12-06T03:52:03.706 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.703+0000 7f7e5159a700 1 -- 172.21.15.132:0/3180665036 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e4400d760 con 0x7f7e540fa0a0 2023-12-06T03:52:03.706 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.703+0000 7f7e5159a700 1 -- 172.21.15.132:0/3180665036 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e4400da30 con 0x7f7e540fa0a0 2023-12-06T03:52:03.706 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.703+0000 7f7e5887e700 1 -- 172.21.15.132:0/3180665036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e540fa0a0 msgr2=0x7f7e540fa4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:03.706 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.703+0000 7f7e5887e700 1 --2- 172.21.15.132:0/3180665036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e540fa0a0 0x7f7e540fa4e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f7e4400b5f0 tx=0x7f7e44006ee0).stop 2023-12-06T03:52:03.706 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.703+0000 7f7e5887e700 1 -- 172.21.15.132:0/3180665036 shutdown_connections 2023-12-06T03:52:03.706 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.703+0000 7f7e5887e700 1 --2- 172.21.15.132:0/3180665036 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e540fa0a0 0x7f7e540fa4e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:03.706 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.703+0000 7f7e5887e700 1 -- 172.21.15.132:0/3180665036 >> 172.21.15.132:0/3180665036 conn(0x7f7e540f5b40 msgr2=0x7f7e540f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:03.707 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.703+0000 7f7e5887e700 1 -- 172.21.15.132:0/3180665036 shutdown_connections 2023-12-06T03:52:03.707 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.703+0000 7f7e5887e700 1 -- 172.21.15.132:0/3180665036 wait complete. 2023-12-06T03:52:03.708 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.704+0000 7f7e5887e700 1 Processor -- start 2023-12-06T03:52:03.708 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.704+0000 7f7e5887e700 1 -- start start 2023-12-06T03:52:03.708 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.704+0000 7f7e5887e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e540fa0a0 0x7f7e5410a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.704+0000 7f7e5887e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e5410a7a0 con 0x7f7e540fa0a0 2023-12-06T03:52:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.704+0000 7f7e5259c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e540fa0a0 0x7f7e5410a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.704+0000 7f7e5259c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e540fa0a0 0x7f7e5410a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:53056/0 (socket says 172.21.15.132:53056) 2023-12-06T03:52:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.704+0000 7f7e5259c700 1 -- 172.21.15.132:0/1691411873 learned_addr learned my addr 172.21.15.132:0/1691411873 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.705+0000 7f7e5259c700 1 -- 172.21.15.132:0/1691411873 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4400b200 con 0x7f7e540fa0a0 2023-12-06T03:52:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.705+0000 7f7e5259c700 1 --2- 172.21.15.132:0/1691411873 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e540fa0a0 0x7f7e5410a2b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f7e44006c70 tx=0x7f7e44016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.705+0000 7f7e42ffd700 1 -- 172.21.15.132:0/1691411873 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e44011b40 con 0x7f7e540fa0a0 2023-12-06T03:52:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.705+0000 7f7e42ffd700 1 -- 172.21.15.132:0/1691411873 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e44011ca0 con 0x7f7e540fa0a0 2023-12-06T03:52:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.705+0000 7f7e42ffd700 1 -- 172.21.15.132:0/1691411873 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e4401d430 con 0x7f7e540fa0a0 2023-12-06T03:52:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.705+0000 7f7e5887e700 1 -- 172.21.15.132:0/1691411873 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e5410a9a0 con 0x7f7e540fa0a0 2023-12-06T03:52:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.705+0000 7f7e5887e700 1 -- 172.21.15.132:0/1691411873 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e5410adb0 con 0x7f7e540fa0a0 2023-12-06T03:52:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.705+0000 7f7e42ffd700 1 -- 172.21.15.132:0/1691411873 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f7e4400f070 con 0x7f7e540fa0a0 2023-12-06T03:52:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.705+0000 7f7e42ffd700 1 -- 172.21.15.132:0/1691411873 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f7e44027460 con 0x7f7e540fa0a0 2023-12-06T03:52:03.709 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.706+0000 7f7e5887e700 1 -- 172.21.15.132:0/1691411873 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e54045ad0 con 0x7f7e540fa0a0 2023-12-06T03:52:03.710 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.708+0000 7f7e42ffd700 1 -- 172.21.15.132:0/1691411873 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f7e44016070 con 0x7f7e540fa0a0 2023-12-06T03:52:03.778 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.774+0000 7f7e5887e700 1 -- 172.21.15.132:0/1691411873 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f7e54103ed0 con 0x7f7e540fa0a0 2023-12-06T03:52:03.778 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.774+0000 7f7e42ffd700 1 -- 172.21.15.132:0/1691411873 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f7e44036070 con 0x7f7e540fa0a0 2023-12-06T03:52:03.778 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: cluster: 2023-12-06T03:52:03.778 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: id: c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:52:03.779 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-12-06T03:52:03.779 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.779 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: services: 2023-12-06T03:52:03.779 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi132 (age 0.450042s) 2023-12-06T03:52:03.779 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: mgr: no daemons active 2023-12-06T03:52:03.779 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-12-06T03:52:03.779 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.779 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: data: 2023-12-06T03:52:03.779 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-12-06T03:52:03.779 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-12-06T03:52:03.779 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-12-06T03:52:03.779 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: pgs: 2023-12-06T03:52:03.779 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.779 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.775+0000 7f7e5887e700 1 -- 172.21.15.132:0/1691411873 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e540fa0a0 msgr2=0x7f7e5410a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:03.780 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.775+0000 7f7e5887e700 1 --2- 172.21.15.132:0/1691411873 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e540fa0a0 0x7f7e5410a2b0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f7e44006c70 tx=0x7f7e44016000).stop 2023-12-06T03:52:03.780 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.775+0000 7f7e5887e700 1 -- 172.21.15.132:0/1691411873 shutdown_connections 2023-12-06T03:52:03.780 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.775+0000 7f7e5887e700 1 --2- 172.21.15.132:0/1691411873 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e540fa0a0 0x7f7e5410a2b0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:03.780 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.775+0000 7f7e5887e700 1 -- 172.21.15.132:0/1691411873 >> 172.21.15.132:0/1691411873 conn(0x7f7e540f5b40 msgr2=0x7f7e540f7840 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:03.780 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.775+0000 7f7e5887e700 1 -- 172.21.15.132:0/1691411873 shutdown_connections 2023-12-06T03:52:03.780 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:03.775+0000 7f7e5887e700 1 -- 172.21.15.132:0/1691411873 wait complete. 2023-12-06T03:52:04.126 INFO:teuthology.orchestra.run.smithi132.stderr:mon is available 2023-12-06T03:52:04.126 INFO:teuthology.orchestra.run.smithi132.stderr:Assimilating anything we can from ceph.conf... 2023-12-06T03:52:04.126 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z -v /var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/mon.smithi132:/var/lib/ceph/mon/ceph-smithi132:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi132/config 2023-12-06T03:52:04.500 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.494+0000 7fdd4923f700 1 Processor -- start 2023-12-06T03:52:04.500 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.494+0000 7fdd4923f700 1 -- start start 2023-12-06T03:52:04.500 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.495+0000 7fdd4923f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd440fc2b0 0x7fdd440fc6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:04.500 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.495+0000 7fdd4923f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd440fcbc0 con 0x7fdd440fc2b0 2023-12-06T03:52:04.500 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.495+0000 7fdd42d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd440fc2b0 0x7fdd440fc6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:04.500 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.495+0000 7fdd42d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd440fc2b0 0x7fdd440fc6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:53070/0 (socket says 172.21.15.132:53070) 2023-12-06T03:52:04.500 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.495+0000 7fdd42d9d700 1 -- 172.21.15.132:0/345230507 learned_addr learned my addr 172.21.15.132:0/345230507 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:04.500 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.495+0000 7fdd42d9d700 1 -- 172.21.15.132:0/345230507 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd440fcd00 con 0x7fdd440fc2b0 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.495+0000 7fdd42d9d700 1 --2- 172.21.15.132:0/345230507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd440fc2b0 0x7fdd440fc6d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fdd3800b5f0 tx=0x7fdd38006ee0).ready entity=mon.0 client_cookie=e75d385f94d0738 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.495+0000 7fdd41d9b700 1 -- 172.21.15.132:0/345230507 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd3800d600 con 0x7fdd440fc2b0 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.495+0000 7fdd41d9b700 1 -- 172.21.15.132:0/345230507 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd3800d760 con 0x7fdd440fc2b0 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.495+0000 7fdd41d9b700 1 -- 172.21.15.132:0/345230507 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd3800da30 con 0x7fdd440fc2b0 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.496+0000 7fdd4923f700 1 -- 172.21.15.132:0/345230507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd440fc2b0 msgr2=0x7fdd440fc6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.496+0000 7fdd4923f700 1 --2- 172.21.15.132:0/345230507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd440fc2b0 0x7fdd440fc6d0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fdd3800b5f0 tx=0x7fdd38006ee0).stop 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.496+0000 7fdd4923f700 1 -- 172.21.15.132:0/345230507 shutdown_connections 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.496+0000 7fdd4923f700 1 --2- 172.21.15.132:0/345230507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd440fc2b0 0x7fdd440fc6d0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.496+0000 7fdd4923f700 1 -- 172.21.15.132:0/345230507 >> 172.21.15.132:0/345230507 conn(0x7fdd440f5b40 msgr2=0x7fdd440f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.496+0000 7fdd4923f700 1 -- 172.21.15.132:0/345230507 shutdown_connections 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.496+0000 7fdd4923f700 1 -- 172.21.15.132:0/345230507 wait complete. 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.497+0000 7fdd4923f700 1 Processor -- start 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.497+0000 7fdd4923f700 1 -- start start 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.497+0000 7fdd4923f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd440fc2b0 0x7fdd4410c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:04.501 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.497+0000 7fdd4923f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4410c960 con 0x7fdd440fc2b0 2023-12-06T03:52:04.502 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.497+0000 7fdd42d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd440fc2b0 0x7fdd4410c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:04.502 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.497+0000 7fdd42d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd440fc2b0 0x7fdd4410c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:53074/0 (socket says 172.21.15.132:53074) 2023-12-06T03:52:04.502 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.497+0000 7fdd42d9d700 1 -- 172.21.15.132:0/4213532645 learned_addr learned my addr 172.21.15.132:0/4213532645 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:04.502 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.497+0000 7fdd42d9d700 1 -- 172.21.15.132:0/4213532645 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3800b200 con 0x7fdd440fc2b0 2023-12-06T03:52:04.502 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.497+0000 7fdd42d9d700 1 --2- 172.21.15.132:0/4213532645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd440fc2b0 0x7fdd4410c470 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fdd38006c70 tx=0x7fdd38016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:04.502 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.497+0000 7fdd2b7fe700 1 -- 172.21.15.132:0/4213532645 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd38011b40 con 0x7fdd440fc2b0 2023-12-06T03:52:04.502 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.497+0000 7fdd2b7fe700 1 -- 172.21.15.132:0/4213532645 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd38011ca0 con 0x7fdd440fc2b0 2023-12-06T03:52:04.502 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.498+0000 7fdd4923f700 1 -- 172.21.15.132:0/4213532645 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd4410cb60 con 0x7fdd440fc2b0 2023-12-06T03:52:04.502 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.498+0000 7fdd2b7fe700 1 -- 172.21.15.132:0/4213532645 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdd3801d430 con 0x7fdd440fc2b0 2023-12-06T03:52:04.502 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.498+0000 7fdd4923f700 1 -- 172.21.15.132:0/4213532645 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4410cf70 con 0x7fdd440fc2b0 2023-12-06T03:52:04.502 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.498+0000 7fdd2b7fe700 1 -- 172.21.15.132:0/4213532645 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fdd3800f070 con 0x7fdd440fc2b0 2023-12-06T03:52:04.502 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.498+0000 7fdd2b7fe700 1 -- 172.21.15.132:0/4213532645 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fdd38027460 con 0x7fdd440fc2b0 2023-12-06T03:52:04.503 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.499+0000 7fdd4923f700 1 -- 172.21.15.132:0/4213532645 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd44045ad0 con 0x7fdd440fc2b0 2023-12-06T03:52:04.503 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.501+0000 7fdd2b7fe700 1 -- 172.21.15.132:0/4213532645 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fdd38016070 con 0x7fdd440fc2b0 2023-12-06T03:52:04.567 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.565+0000 7fdd4923f700 1 -- 172.21.15.132:0/4213532645 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fdd441015d0 con 0x7fdd440fc2b0 2023-12-06T03:52:04.572 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.568+0000 7fdd2b7fe700 1 -- 172.21.15.132:0/4213532645 <== mon.0 v2:172.21.15.132:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fdd3801d590 con 0x7fdd440fc2b0 2023-12-06T03:52:04.572 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.568+0000 7fdd2b7fe700 1 -- 172.21.15.132:0/4213532645 <== mon.0 v2:172.21.15.132:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+466 (secure 0 0 0) 0x7fdd380276f0 con 0x7fdd440fc2b0 2023-12-06T03:52:04.572 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: [global] 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: fsid = c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.132:3300,v1:172.21.15.132:6789] 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: [mgr] 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: [osd] 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-12-06T03:52:04.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-12-06T03:52:04.574 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-12-06T03:52:04.574 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-12-06T03:52:04.574 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-12-06T03:52:04.574 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.569+0000 7fdd4923f700 1 -- 172.21.15.132:0/4213532645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd440fc2b0 msgr2=0x7fdd4410c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:04.574 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.569+0000 7fdd4923f700 1 --2- 172.21.15.132:0/4213532645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd440fc2b0 0x7fdd4410c470 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fdd38006c70 tx=0x7fdd38016000).stop 2023-12-06T03:52:04.574 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.569+0000 7fdd4923f700 1 -- 172.21.15.132:0/4213532645 shutdown_connections 2023-12-06T03:52:04.574 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.569+0000 7fdd4923f700 1 --2- 172.21.15.132:0/4213532645 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdd440fc2b0 0x7fdd4410c470 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:04.574 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.569+0000 7fdd4923f700 1 -- 172.21.15.132:0/4213532645 >> 172.21.15.132:0/4213532645 conn(0x7fdd440f5b40 msgr2=0x7fdd440f7870 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:04.574 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.569+0000 7fdd4923f700 1 -- 172.21.15.132:0/4213532645 shutdown_connections 2023-12-06T03:52:04.574 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:04.569+0000 7fdd4923f700 1 -- 172.21.15.132:0/4213532645 wait complete. 2023-12-06T03:52:04.592 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:04 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111290]: cluster 2023-12-06T03:52:03.325894+0000 mon.smithi132 (mon.0) 2 : cluster [INF] mon.smithi132 is new leader, mons smithi132 in quorum (ranks 0) 2023-12-06T03:52:04.592 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:04 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111290]: cluster 2023-12-06T03:52:03.326169+0000 mon.smithi132 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi132=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0]} 2023-12-06T03:52:04.592 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:04 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111290]: cluster 2023-12-06T03:52:03.329193+0000 mon.smithi132 (mon.0) 4 : cluster [DBG] fsmap 2023-12-06T03:52:04.592 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:04 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111290]: cluster 2023-12-06T03:52:03.333583+0000 mon.smithi132 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-06T03:52:04.592 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:04 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111290]: cluster 2023-12-06T03:52:03.334232+0000 mon.smithi132 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-12-06T03:52:04.592 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:04 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111290]: audit 2023-12-06T03:52:03.775904+0000 mon.smithi132 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.132:0/1691411873' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-12-06T03:52:04.917 INFO:teuthology.orchestra.run.smithi132.stderr:Generating new minimal ceph.conf... 2023-12-06T03:52:04.917 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z -v /var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/mon.smithi132:/var/lib/ceph/mon/ceph-smithi132:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi132/config 2023-12-06T03:52:05.265 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.261+0000 7f1686fb6700 1 Processor -- start 2023-12-06T03:52:05.265 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.261+0000 7f1686fb6700 1 -- start start 2023-12-06T03:52:05.265 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.262+0000 7f1686fb6700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1680070430 0x7f16800fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:05.265 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.262+0000 7f1686fb6700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16800fedd0 con 0x7f1680070430 2023-12-06T03:52:05.265 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.262+0000 7f1684d52700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1680070430 0x7f16800fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:05.265 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.262+0000 7f1684d52700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1680070430 0x7f16800fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:53088/0 (socket says 172.21.15.132:53088) 2023-12-06T03:52:05.265 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.262+0000 7f1684d52700 1 -- 172.21.15.132:0/4032035269 learned_addr learned my addr 172.21.15.132:0/4032035269 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:05.266 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.262+0000 7f1684d52700 1 -- 172.21.15.132:0/4032035269 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16800fef10 con 0x7f1680070430 2023-12-06T03:52:05.266 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.262+0000 7f1684d52700 1 --2- 172.21.15.132:0/4032035269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1680070430 0x7f16800fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f167400b5f0 tx=0x7f1674006ee0).ready entity=mon.0 client_cookie=57ab25db6bd9a479 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:05.266 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.262+0000 7f167f7fe700 1 -- 172.21.15.132:0/4032035269 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167400d600 con 0x7f1680070430 2023-12-06T03:52:05.266 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.262+0000 7f167f7fe700 1 -- 172.21.15.132:0/4032035269 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f167400dbd0 con 0x7f1680070430 2023-12-06T03:52:05.266 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.262+0000 7f167f7fe700 1 -- 172.21.15.132:0/4032035269 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1674011c00 con 0x7f1680070430 2023-12-06T03:52:05.266 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.263+0000 7f1686fb6700 1 -- 172.21.15.132:0/4032035269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1680070430 msgr2=0x7f16800fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:05.266 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.263+0000 7f1686fb6700 1 --2- 172.21.15.132:0/4032035269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1680070430 0x7f16800fe8e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f167400b5f0 tx=0x7f1674006ee0).stop 2023-12-06T03:52:05.266 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.263+0000 7f1686fb6700 1 -- 172.21.15.132:0/4032035269 shutdown_connections 2023-12-06T03:52:05.266 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.263+0000 7f1686fb6700 1 --2- 172.21.15.132:0/4032035269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1680070430 0x7f16800fe8e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:05.266 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.263+0000 7f1686fb6700 1 -- 172.21.15.132:0/4032035269 >> 172.21.15.132:0/4032035269 conn(0x7f16800f63c0 msgr2=0x7f16800f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:05.267 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.263+0000 7f1686fb6700 1 -- 172.21.15.132:0/4032035269 shutdown_connections 2023-12-06T03:52:05.267 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.263+0000 7f1686fb6700 1 -- 172.21.15.132:0/4032035269 wait complete. 2023-12-06T03:52:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.264+0000 7f1686fb6700 1 Processor -- start 2023-12-06T03:52:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.264+0000 7f1686fb6700 1 -- start start 2023-12-06T03:52:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.264+0000 7f1686fb6700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1680070430 0x7f16800fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.264+0000 7f1686fb6700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16800fa7e0 con 0x7f1680070430 2023-12-06T03:52:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.264+0000 7f1684d52700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1680070430 0x7f16800fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.264+0000 7f1684d52700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1680070430 0x7f16800fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:53104/0 (socket says 172.21.15.132:53104) 2023-12-06T03:52:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.264+0000 7f1684d52700 1 -- 172.21.15.132:0/2577845044 learned_addr learned my addr 172.21.15.132:0/2577845044 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.265+0000 7f1684d52700 1 -- 172.21.15.132:0/2577845044 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167400b200 con 0x7f1680070430 2023-12-06T03:52:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.265+0000 7f1684d52700 1 --2- 172.21.15.132:0/2577845044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1680070430 0x7f16800fc360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f167400e040 tx=0x7f1674006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.265+0000 7f167d7fa700 1 -- 172.21.15.132:0/2577845044 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167400d9f0 con 0x7f1680070430 2023-12-06T03:52:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.265+0000 7f167d7fa700 1 -- 172.21.15.132:0/2577845044 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f167401a9f0 con 0x7f1680070430 2023-12-06T03:52:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.265+0000 7f1686fb6700 1 -- 172.21.15.132:0/2577845044 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16800fa980 con 0x7f1680070430 2023-12-06T03:52:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.265+0000 7f167d7fa700 1 -- 172.21.15.132:0/2577845044 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167401b8c0 con 0x7f1680070430 2023-12-06T03:52:05.268 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.265+0000 7f1686fb6700 1 -- 172.21.15.132:0/2577845044 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16800fad90 con 0x7f1680070430 2023-12-06T03:52:05.269 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.265+0000 7f167d7fa700 1 -- 172.21.15.132:0/2577845044 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f167400f070 con 0x7f1680070430 2023-12-06T03:52:05.269 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.265+0000 7f167d7fa700 1 -- 172.21.15.132:0/2577845044 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1674025b30 con 0x7f1680070430 2023-12-06T03:52:05.269 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.266+0000 7f1686fb6700 1 -- 172.21.15.132:0/2577845044 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1680045ad0 con 0x7f1680070430 2023-12-06T03:52:05.270 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.268+0000 7f167d7fa700 1 -- 172.21.15.132:0/2577845044 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1674016070 con 0x7f1680070430 2023-12-06T03:52:05.334 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.332+0000 7f1686fb6700 1 -- 172.21.15.132:0/2577845044 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f1680104820 con 0x7f1680070430 2023-12-06T03:52:05.334 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.333+0000 7f167d7fa700 1 -- 172.21.15.132:0/2577845044 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f167401bd20 con 0x7f1680070430 2023-12-06T03:52:05.336 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.333+0000 7f1686fb6700 1 -- 172.21.15.132:0/2577845044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1680070430 msgr2=0x7f16800fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:05.336 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.333+0000 7f1686fb6700 1 --2- 172.21.15.132:0/2577845044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1680070430 0x7f16800fc360 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f167400e040 tx=0x7f1674006ee0).stop 2023-12-06T03:52:05.336 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.333+0000 7f1686fb6700 1 -- 172.21.15.132:0/2577845044 shutdown_connections 2023-12-06T03:52:05.336 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.333+0000 7f1686fb6700 1 --2- 172.21.15.132:0/2577845044 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1680070430 0x7f16800fc360 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:05.336 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.333+0000 7f1686fb6700 1 -- 172.21.15.132:0/2577845044 >> 172.21.15.132:0/2577845044 conn(0x7f16800f63c0 msgr2=0x7f16800f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:05.336 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.333+0000 7f1686fb6700 1 -- 172.21.15.132:0/2577845044 shutdown_connections 2023-12-06T03:52:05.336 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:05.333+0000 7f1686fb6700 1 -- 172.21.15.132:0/2577845044 wait complete. 2023-12-06T03:52:05.603 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111290]: audit 2023-12-06T03:52:04.567042+0000 mon.smithi132 (mon.0) 8 : audit [INF] from='client.? 172.21.15.132:0/4213532645' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-06T03:52:05.603 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111290]: audit 2023-12-06T03:52:04.570206+0000 mon.smithi132 (mon.0) 9 : audit [INF] from='client.? 172.21.15.132:0/4213532645' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-06T03:52:05.603 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111290]: audit 2023-12-06T03:52:05.334291+0000 mon.smithi132 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.132:0/2577845044' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-06T03:52:05.685 INFO:teuthology.orchestra.run.smithi132.stderr:Restarting the monitor... 2023-12-06T03:52:05.685 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl restart ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi132 2023-12-06T03:52:05.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:05 smithi132 systemd[1]: Stopping Ceph mon.smithi132 for c121c6d8-93ea-11ee-95a2-87774f69a715... 2023-12-06T03:52:05.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111290]: debug 2023-12-06T03:52:05.764+0000 7f01c6677700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi132 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-06T03:52:05.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111290]: debug 2023-12-06T03:52:05.764+0000 7f01c6677700 -1 mon.smithi132@0(leader) e1 *** Got Signal Terminated *** 2023-12-06T03:52:06.356 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 podman[111611]: ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132 2023-12-06T03:52:06.357 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 systemd[1]: ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi132.service: Succeeded. 2023-12-06T03:52:06.357 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 systemd[1]: Stopped Ceph mon.smithi132 for c121c6d8-93ea-11ee-95a2-87774f69a715. 2023-12-06T03:52:06.357 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 systemd[1]: Starting Ceph mon.smithi132 for c121c6d8-93ea-11ee-95a2-87774f69a715... 2023-12-06T03:52:06.468 INFO:teuthology.orchestra.run.smithi132.stderr:Setting mon public_network to 172.21.0.0/20 2023-12-06T03:52:06.469 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-12-06T03:52:06.838 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.834+0000 7f8704464700 1 Processor -- start 2023-12-06T03:52:06.838 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.834+0000 7f8704464700 1 -- start start 2023-12-06T03:52:06.839 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.835+0000 7f8704464700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86fc070430 0x7f86fc0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:06.839 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.835+0000 7f8704464700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86fc0fedd0 con 0x7f86fc070430 2023-12-06T03:52:06.839 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.835+0000 7f8702200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86fc070430 0x7f86fc0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:06.839 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.835+0000 7f8702200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86fc070430 0x7f86fc0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:53116/0 (socket says 172.21.15.132:53116) 2023-12-06T03:52:06.839 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.835+0000 7f8702200700 1 -- 172.21.15.132:0/2199594021 learned_addr learned my addr 172.21.15.132:0/2199594021 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:06.839 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.836+0000 7f8702200700 1 -- 172.21.15.132:0/2199594021 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86fc0fef10 con 0x7f86fc070430 2023-12-06T03:52:06.839 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.836+0000 7f8702200700 1 --2- 172.21.15.132:0/2199594021 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86fc070430 0x7f86fc0fe8e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f86f800b5f0 tx=0x7f86f8006ee0).ready entity=mon.0 client_cookie=eca10b764f6eb89d server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:06.839 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.836+0000 7f87011fe700 1 -- 172.21.15.132:0/2199594021 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86f800d600 con 0x7f86fc070430 2023-12-06T03:52:06.839 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.836+0000 7f87011fe700 1 -- 172.21.15.132:0/2199594021 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f86f800dbd0 con 0x7f86fc070430 2023-12-06T03:52:06.839 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.836+0000 7f87011fe700 1 -- 172.21.15.132:0/2199594021 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86f8011c00 con 0x7f86fc070430 2023-12-06T03:52:06.839 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.837+0000 7f8704464700 1 -- 172.21.15.132:0/2199594021 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86fc070430 msgr2=0x7f86fc0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:06.839 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.837+0000 7f8704464700 1 --2- 172.21.15.132:0/2199594021 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86fc070430 0x7f86fc0fe8e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f86f800b5f0 tx=0x7f86f8006ee0).stop 2023-12-06T03:52:06.841 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.837+0000 7f8704464700 1 -- 172.21.15.132:0/2199594021 shutdown_connections 2023-12-06T03:52:06.841 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.837+0000 7f8704464700 1 --2- 172.21.15.132:0/2199594021 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86fc070430 0x7f86fc0fe8e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:06.841 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.837+0000 7f8704464700 1 -- 172.21.15.132:0/2199594021 >> 172.21.15.132:0/2199594021 conn(0x7f86fc0f63c0 msgr2=0x7f86fc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:06.841 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.837+0000 7f8704464700 1 -- 172.21.15.132:0/2199594021 shutdown_connections 2023-12-06T03:52:06.842 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.837+0000 7f8704464700 1 -- 172.21.15.132:0/2199594021 wait complete. 2023-12-06T03:52:06.842 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.837+0000 7f8704464700 1 Processor -- start 2023-12-06T03:52:06.842 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.838+0000 7f8704464700 1 -- start start 2023-12-06T03:52:06.842 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.838+0000 7f8704464700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86fc070430 0x7f86fc10aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:06.842 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.838+0000 7f8704464700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86fc10aef0 con 0x7f86fc070430 2023-12-06T03:52:06.842 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.838+0000 7f8702200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86fc070430 0x7f86fc10aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:06.842 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.838+0000 7f8702200700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86fc070430 0x7f86fc10aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:53126/0 (socket says 172.21.15.132:53126) 2023-12-06T03:52:06.842 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.838+0000 7f8702200700 1 -- 172.21.15.132:0/1388855908 learned_addr learned my addr 172.21.15.132:0/1388855908 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:06.842 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.838+0000 7f8702200700 1 -- 172.21.15.132:0/1388855908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f800b200 con 0x7f86fc070430 2023-12-06T03:52:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.838+0000 7f8702200700 1 --2- 172.21.15.132:0/1388855908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86fc070430 0x7f86fc10aa00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f86f800b190 tx=0x7f86f8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.838+0000 7f86eaffd700 1 -- 172.21.15.132:0/1388855908 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86f801b410 con 0x7f86fc070430 2023-12-06T03:52:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.838+0000 7f86eaffd700 1 -- 172.21.15.132:0/1388855908 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f86f801b9e0 con 0x7f86fc070430 2023-12-06T03:52:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.838+0000 7f8704464700 1 -- 172.21.15.132:0/1388855908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86fc10b0f0 con 0x7f86fc070430 2023-12-06T03:52:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.838+0000 7f86eaffd700 1 -- 172.21.15.132:0/1388855908 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86f801a880 con 0x7f86fc070430 2023-12-06T03:52:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.838+0000 7f8704464700 1 -- 172.21.15.132:0/1388855908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86fc10b500 con 0x7f86fc070430 2023-12-06T03:52:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.839+0000 7f86eaffd700 1 -- 172.21.15.132:0/1388855908 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f86f800f070 con 0x7f86fc070430 2023-12-06T03:52:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.839+0000 7f86eaffd700 1 -- 172.21.15.132:0/1388855908 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f86f8025b70 con 0x7f86fc070430 2023-12-06T03:52:06.843 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.839+0000 7f8704464700 1 -- 172.21.15.132:0/1388855908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86fc045ad0 con 0x7f86fc070430 2023-12-06T03:52:06.844 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.842+0000 7f86eaffd700 1 -- 172.21.15.132:0/1388855908 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f86f8016070 con 0x7f86fc070430 2023-12-06T03:52:06.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 bash[111683]: eaa24294ba16fa6462f96501238e959420d64335aeb1bff3fa05c5c2b5f8480c 2023-12-06T03:52:06.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 systemd[1]: Started Ceph mon.smithi132 for c121c6d8-93ea-11ee-95a2-87774f69a715. 2023-12-06T03:52:06.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.465+0000 7f5c41daa700 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-06T03:52:06.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.465+0000 7f5c41daa700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-12-06T03:52:06.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.465+0000 7f5c41daa700 0 pidfile_write: ignore empty --pid-file 2023-12-06T03:52:06.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.470+0000 7f5c41daa700 0 load: jerasure load: lrc load: isa 2023-12-06T03:52:06.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: RocksDB version: 6.8.1 2023-12-06T03:52:06.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-06T03:52:06.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Compile date May 13 2021 2023-12-06T03:52:06.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: DB SUMMARY 2023-12-06T03:52:06.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: CURRENT file: CURRENT 2023-12-06T03:52:06.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: IDENTITY file: IDENTITY 2023-12-06T03:52:06.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi132/store.db dir, Total Num: 1, files: 000004.sst 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi132/store.db: 000006.log size: 93460 ; 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.error_if_exists: 0 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.create_if_missing: 0 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.paranoid_checks: 1 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.env: 0x55e79ea7a1c0 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.fs: Posix File System 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.info_log: 0x55e79f439960 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.statistics: (nil) 2023-12-06T03:52:06.859 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.use_fsync: 0 2023-12-06T03:52:06.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_log_file_size: 0 2023-12-06T03:52:06.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-06T03:52:06.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-06T03:52:06.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-06T03:52:06.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-06T03:52:06.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.allow_fallocate: 1 2023-12-06T03:52:06.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-06T03:52:06.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-06T03:52:06.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.use_direct_reads: 0 2023-12-06T03:52:06.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-06T03:52:06.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.create_missing_column_families: 0 2023-12-06T03:52:06.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.db_log_dir: 2023-12-06T03:52:06.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi132/store.db 2023-12-06T03:52:06.860 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-06T03:52:06.862 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_subcompactions: 1 2023-12-06T03:52:06.862 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_background_flushes: -1 2023-12-06T03:52:06.862 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-06T03:52:06.862 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.advise_random_on_open: 1 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.write_buffer_manager: 0x55e79f43ffb0 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.rate_limiter: (nil) 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-06T03:52:06.863 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.unordered_write: 0 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.row_cache: None 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.wal_filter: None 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.preserve_deletes: 0 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.two_write_queues: 0 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.manual_wal_flush: 0 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.atomic_flush: 0 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-06T03:52:06.864 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.log_readahead_size: 0 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_background_jobs: 2 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_background_compactions: -1 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_total_wal_size: 0 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_open_files: -1 2023-12-06T03:52:06.865 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.bytes_per_sync: 0 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Compression algorithms supported: 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: kZSTD supported: 0 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: kXpressCompression supported: 0 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: kLZ4Compression supported: 1 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: kBZip2Compression supported: 0 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: kZlibCompression supported: 1 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: kSnappyCompression supported: 1 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi132/store.db/MANIFEST-000005 2023-12-06T03:52:06.866 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.merge_operator: 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compaction_filter: None 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compaction_filter_factory: None 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e79f350128) 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cache_index_and_filter_blocks: 1 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: pin_top_level_index_and_filter: 1 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: index_type: 0 2023-12-06T03:52:06.867 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: data_block_index_type: 0 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: index_shortening: 1 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: data_block_hash_table_util_ratio: 0.750000 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: hash_index_allow_collision: 1 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: checksum: 1 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: no_block_cache: 0 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: block_cache: 0x55e79f386f10 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: block_cache_name: BinnedLRUCache 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: block_cache_options: 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: capacity : 536870912 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: num_shard_bits : 4 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: strict_capacity_limit : 0 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: high_pri_pool_ratio: 0.000 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: block_cache_compressed: (nil) 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: persistent_cache: (nil) 2023-12-06T03:52:06.868 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: block_size: 4096 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: block_size_deviation: 10 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: block_restart_interval: 16 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: index_block_restart_interval: 1 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: metadata_block_size: 4096 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: partition_filters: 0 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: use_delta_encoding: 1 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: whole_key_filtering: 1 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: verify_compression: 0 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: read_amp_bytes_per_bit: 0 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: format_version: 2 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: enable_index_compression: 1 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: block_align: 0 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.869 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compression: NoCompression 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.num_levels: 7 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-06T03:52:06.870 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compression_opts.level: 32767 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compression_opts.enabled: false 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-06T03:52:06.871 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-06T03:52:06.872 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-06T03:52:06.872 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-06T03:52:06.872 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-06T03:52:06.872 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-06T03:52:06.872 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-06T03:52:06.872 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-06T03:52:06.872 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-06T03:52:06.872 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-06T03:52:06.872 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.arena_block_size: 4194304 2023-12-06T03:52:06.872 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-06T03:52:06.872 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.table_properties_collectors: 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.inplace_update_support: 0 2023-12-06T03:52:06.873 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-06T03:52:06.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-06T03:52:06.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-06T03:52:06.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-06T03:52:06.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.bloom_locality: 0 2023-12-06T03:52:06.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.max_successive_merges: 0 2023-12-06T03:52:06.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-06T03:52:06.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-06T03:52:06.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.force_consistency_checks: 0 2023-12-06T03:52:06.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-06T03:52:06.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.ttl: 2592000 2023-12-06T03:52:06.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.471+0000 7f5c41daa700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-06T03:52:06.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.472+0000 7f5c41daa700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi132/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-06T03:52:06.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.874 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.472+0000 7f5c41daa700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-12-06T03:52:06.875 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.875 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.472+0000 7f5c41daa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701834726473702, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-12-06T03:52:06.875 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.472+0000 7f5c41daa700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-12-06T03:52:06.875 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.472+0000 7f5c41daa700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-06T03:52:06.875 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.473+0000 7f5c41daa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701834726474984, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 89240, "table_properties": {"data_size": 87355, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 80496, "raw_average_value_size": 323, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1701834726, "oldest_key_time": 3, "file_creation_time": 0}} 2023-12-06T03:52:06.875 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.473+0000 7f5c41daa700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-12-06T03:52:06.875 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.875 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.474+0000 7f5c41daa700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1701834726476252, "job": 1, "event": "recovery_finished"} 2023-12-06T03:52:06.875 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.475+0000 7f5c41daa700 4 rocksdb: DB pointer 0x55e79f44b800 2023-12-06T03:52:06.875 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.476+0000 7f5c28bc2700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-12-06T03:52:06.875 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.476+0000 7f5c28bc2700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: ** DB Stats ** 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: ** Compaction Stats [default] ** 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: L0 2/0 88.84 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 97.8 0.00 0.00 1 0.001 0 0 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Sum 2/0 88.84 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 97.8 0.00 0.00 1 0.001 0 0 2023-12-06T03:52:06.876 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 97.8 0.00 0.00 1 0.001 0 0 2023-12-06T03:52:06.877 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.877 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: ** Compaction Stats [default] ** 2023-12-06T03:52:06.877 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-06T03:52:06.877 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-06T03:52:06.877 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 97.8 0.00 0.00 1 0.001 0 0 2023-12-06T03:52:06.877 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-06T03:52:06.877 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-06T03:52:06.877 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-06T03:52:06.877 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: AddFile(Total Files): cumulative 0, interval 0 2023-12-06T03:52:06.877 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-06T03:52:06.877 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: AddFile(Keys): cumulative 0, interval 0 2023-12-06T03:52:06.877 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Cumulative compaction: 0.00 GB write, 18.86 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-06T03:52:06.877 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Interval compaction: 0.00 GB write, 18.86 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-06T03:52:06.877 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: ** File Read Latency Histogram By Level [default] ** 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: ** Compaction Stats [default] ** 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: L0 2/0 88.84 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 97.8 0.00 0.00 1 0.001 0 0 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Sum 2/0 88.84 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 97.8 0.00 0.00 1 0.001 0 0 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: ** Compaction Stats [default] ** 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 97.8 0.00 0.00 1 0.001 0 0 2023-12-06T03:52:06.878 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: AddFile(Total Files): cumulative 0, interval 0 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: AddFile(Keys): cumulative 0, interval 0 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Cumulative compaction: 0.00 GB write, 18.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: ** File Read Latency Histogram By Level [default] ** 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.476+0000 7f5c41daa700 0 starting mon.smithi132 rank 0 at public addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] at bind addrs [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi132 fsid c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.478+0000 7f5c41daa700 1 mon.smithi132@-1(???) e1 preinit fsid c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.478+0000 7f5c41daa700 0 mon.smithi132@-1(???).mds e1 new map 2023-12-06T03:52:06.879 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.478+0000 7f5c41daa700 0 mon.smithi132@-1(???).mds e1 print_map 2023-12-06T03:52:06.880 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: e1 2023-12-06T03:52:06.880 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: enable_multiple, ever_enabled_multiple: 1,1 2023-12-06T03:52:06.880 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-06T03:52:06.880 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: legacy client fscid: -1 2023-12-06T03:52:06.880 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.880 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: No filesystems configured 2023-12-06T03:52:06.880 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: 2023-12-06T03:52:06.880 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.478+0000 7f5c41daa700 0 mon.smithi132@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-06T03:52:06.880 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.478+0000 7f5c41daa700 0 mon.smithi132@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-06T03:52:06.880 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.478+0000 7f5c41daa700 0 mon.smithi132@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-06T03:52:06.880 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.478+0000 7f5c41daa700 0 mon.smithi132@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-06T03:52:06.880 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:04.567042+0000 mon.smithi132 (mon.0) 8 : audit [INF] from='client.? 172.21.15.132:0/4213532645' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-06T03:52:06.880 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:04.570206+0000 mon.smithi132 (mon.0) 9 : audit [INF] from='client.? 172.21.15.132:0/4213532645' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-06T03:52:06.880 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:05.334291+0000 mon.smithi132 (mon.0) 10 : audit [DBG] from='client.? 172.21.15.132:0/2577845044' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-06T03:52:06.881 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T03:52:06.478+0000 7f5c41daa700 1 mon.smithi132@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-12-06T03:52:06.881 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:06.497092+0000 mon.smithi132 (mon.0) 1 : cluster [INF] mon.smithi132 is new leader, mons smithi132 in quorum (ranks 0) 2023-12-06T03:52:06.881 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:06.497159+0000 mon.smithi132 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi132=[v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0]} 2023-12-06T03:52:06.881 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:06.497198+0000 mon.smithi132 (mon.0) 3 : cluster [DBG] fsmap 2023-12-06T03:52:06.881 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:06.497222+0000 mon.smithi132 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-06T03:52:06.881 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:06.497646+0000 mon.smithi132 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-12-06T03:52:06.908 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.906+0000 7f8704464700 1 -- 172.21.15.132:0/1388855908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f86fc104860 con 0x7f86fc070430 2023-12-06T03:52:06.911 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.909+0000 7f86eaffd700 1 -- 172.21.15.132:0/1388855908 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f86f8025db0 con 0x7f86fc070430 2023-12-06T03:52:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.910+0000 7f8704464700 1 -- 172.21.15.132:0/1388855908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86fc070430 msgr2=0x7f86fc10aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.910+0000 7f8704464700 1 --2- 172.21.15.132:0/1388855908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86fc070430 0x7f86fc10aa00 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f86f800b190 tx=0x7f86f8006ee0).stop 2023-12-06T03:52:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.910+0000 7f8704464700 1 -- 172.21.15.132:0/1388855908 shutdown_connections 2023-12-06T03:52:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.910+0000 7f8704464700 1 --2- 172.21.15.132:0/1388855908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86fc070430 0x7f86fc10aa00 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.910+0000 7f8704464700 1 -- 172.21.15.132:0/1388855908 >> 172.21.15.132:0/1388855908 conn(0x7f86fc0f63c0 msgr2=0x7f86fc0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.910+0000 7f8704464700 1 -- 172.21.15.132:0/1388855908 shutdown_connections 2023-12-06T03:52:06.912 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:06.910+0000 7f8704464700 1 -- 172.21.15.132:0/1388855908 wait complete. 2023-12-06T03:52:07.285 INFO:teuthology.orchestra.run.smithi132.stderr:Wrote config to /etc/ceph/ceph.conf 2023-12-06T03:52:07.285 INFO:teuthology.orchestra.run.smithi132.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-12-06T03:52:07.285 INFO:teuthology.orchestra.run.smithi132.stderr:Creating mgr... 2023-12-06T03:52:07.286 INFO:teuthology.orchestra.run.smithi132.stderr:Verifying port 9283 ... 2023-12-06T03:52:07.287 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl daemon-reload 2023-12-06T03:52:07.380 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl stop ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mgr.smithi132.gokxja 2023-12-06T03:52:07.403 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl reset-failed ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mgr.smithi132.gokxja 2023-12-06T03:52:07.409 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: Failed to reset failed state of unit ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mgr.smithi132.gokxja.service: Unit ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mgr.smithi132.gokxja.service not loaded. 2023-12-06T03:52:07.409 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl enable ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mgr.smithi132.gokxja 2023-12-06T03:52:07.415 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: Created symlink /etc/systemd/system/ceph-c121c6d8-93ea-11ee-95a2-87774f69a715.target.wants/ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mgr.smithi132.gokxja.service → /etc/systemd/system/ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@.service. 2023-12-06T03:52:07.497 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl start ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mgr.smithi132.gokxja 2023-12-06T03:52:07.819 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-06T03:52:07.826 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: disabled 2023-12-06T03:52:07.827 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-active firewalld.service 2023-12-06T03:52:07.834 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: inactive 2023-12-06T03:52:07.835 INFO:teuthology.orchestra.run.smithi132.stderr:firewalld.service is not enabled 2023-12-06T03:52:07.835 INFO:teuthology.orchestra.run.smithi132.stderr:Not possible to enable service . firewalld.service is not available 2023-12-06T03:52:07.835 INFO:teuthology.orchestra.run.smithi132.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-12-06T03:52:07.835 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-06T03:52:07.842 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: disabled 2023-12-06T03:52:07.843 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-active firewalld.service 2023-12-06T03:52:07.850 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: inactive 2023-12-06T03:52:07.851 INFO:teuthology.orchestra.run.smithi132.stderr:firewalld.service is not enabled 2023-12-06T03:52:07.851 INFO:teuthology.orchestra.run.smithi132.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-12-06T03:52:07.851 INFO:teuthology.orchestra.run.smithi132.stderr:Waiting for mgr to start... 2023-12-06T03:52:07.851 INFO:teuthology.orchestra.run.smithi132.stderr:Waiting for mgr... 2023-12-06T03:52:07.851 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-06T03:52:08.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:07 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:06.910812+0000 mon.smithi132 (mon.0) 6 : audit [INF] from='client.? 172.21.15.132:0/1388855908' entity='client.admin' 2023-12-06T03:52:08.207 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.204+0000 7fc2f3e29700 1 Processor -- start 2023-12-06T03:52:08.209 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.205+0000 7fc2f3e29700 1 -- start start 2023-12-06T03:52:08.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.205+0000 7fc2f3e29700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec0703f0 0x7fc2ec0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:08.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.205+0000 7fc2f3e29700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec0fee10 con 0x7fc2ec0703f0 2023-12-06T03:52:08.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.205+0000 7fc2f1bc5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec0703f0 0x7fc2ec0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:08.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.205+0000 7fc2f1bc5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec0703f0 0x7fc2ec0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49854/0 (socket says 172.21.15.132:49854) 2023-12-06T03:52:08.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.205+0000 7fc2f1bc5700 1 -- 172.21.15.132:0/1016126170 learned_addr learned my addr 172.21.15.132:0/1016126170 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:08.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.206+0000 7fc2f1bc5700 1 -- 172.21.15.132:0/1016126170 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2ec0fef50 con 0x7fc2ec0703f0 2023-12-06T03:52:08.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.206+0000 7fc2f1bc5700 1 --2- 172.21.15.132:0/1016126170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec0703f0 0x7fc2ec0fe920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fc2e800b5f0 tx=0x7fc2e8006ee0).ready entity=mon.0 client_cookie=2d43d4964b6528bc server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:08.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.206+0000 7fc2f0bc3700 1 -- 172.21.15.132:0/1016126170 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e800d600 con 0x7fc2ec0703f0 2023-12-06T03:52:08.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.206+0000 7fc2f0bc3700 1 -- 172.21.15.132:0/1016126170 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc2e800dbd0 con 0x7fc2ec0703f0 2023-12-06T03:52:08.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.206+0000 7fc2f0bc3700 1 -- 172.21.15.132:0/1016126170 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e8011c00 con 0x7fc2ec0703f0 2023-12-06T03:52:08.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.206+0000 7fc2f3e29700 1 -- 172.21.15.132:0/1016126170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec0703f0 msgr2=0x7fc2ec0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:08.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.206+0000 7fc2f3e29700 1 --2- 172.21.15.132:0/1016126170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec0703f0 0x7fc2ec0fe920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fc2e800b5f0 tx=0x7fc2e8006ee0).stop 2023-12-06T03:52:08.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.207+0000 7fc2f3e29700 1 -- 172.21.15.132:0/1016126170 shutdown_connections 2023-12-06T03:52:08.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.207+0000 7fc2f3e29700 1 --2- 172.21.15.132:0/1016126170 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec0703f0 0x7fc2ec0fe920 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:08.211 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.207+0000 7fc2f3e29700 1 -- 172.21.15.132:0/1016126170 >> 172.21.15.132:0/1016126170 conn(0x7fc2ec0f63c0 msgr2=0x7fc2ec0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:08.211 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.207+0000 7fc2f3e29700 1 -- 172.21.15.132:0/1016126170 shutdown_connections 2023-12-06T03:52:08.211 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.207+0000 7fc2f3e29700 1 -- 172.21.15.132:0/1016126170 wait complete. 2023-12-06T03:52:08.211 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.207+0000 7fc2f3e29700 1 Processor -- start 2023-12-06T03:52:08.211 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.207+0000 7fc2f3e29700 1 -- start start 2023-12-06T03:52:08.211 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.207+0000 7fc2f3e29700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec0703f0 0x7fc2ec10aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:08.211 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.207+0000 7fc2f3e29700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec10af40 con 0x7fc2ec0703f0 2023-12-06T03:52:08.211 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.208+0000 7fc2f1bc5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec0703f0 0x7fc2ec10aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:08.211 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.208+0000 7fc2f1bc5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec0703f0 0x7fc2ec10aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49868/0 (socket says 172.21.15.132:49868) 2023-12-06T03:52:08.211 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.208+0000 7fc2f1bc5700 1 -- 172.21.15.132:0/3880157511 learned_addr learned my addr 172.21.15.132:0/3880157511 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:08.211 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.208+0000 7fc2f1bc5700 1 -- 172.21.15.132:0/3880157511 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e800b200 con 0x7fc2ec0703f0 2023-12-06T03:52:08.212 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.208+0000 7fc2f1bc5700 1 --2- 172.21.15.132:0/3880157511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec0703f0 0x7fc2ec10aa50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc2e800e040 tx=0x7fc2e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:08.212 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.208+0000 7fc2da7fc700 1 -- 172.21.15.132:0/3880157511 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e800d9f0 con 0x7fc2ec0703f0 2023-12-06T03:52:08.212 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.208+0000 7fc2da7fc700 1 -- 172.21.15.132:0/3880157511 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc2e801a9f0 con 0x7fc2ec0703f0 2023-12-06T03:52:08.212 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.208+0000 7fc2da7fc700 1 -- 172.21.15.132:0/3880157511 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e801b8c0 con 0x7fc2ec0703f0 2023-12-06T03:52:08.212 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.208+0000 7fc2f3e29700 1 -- 172.21.15.132:0/3880157511 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2ec10b140 con 0x7fc2ec0703f0 2023-12-06T03:52:08.213 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.208+0000 7fc2f3e29700 1 -- 172.21.15.132:0/3880157511 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ec10b550 con 0x7fc2ec0703f0 2023-12-06T03:52:08.213 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.209+0000 7fc2da7fc700 1 -- 172.21.15.132:0/3880157511 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fc2e800f070 con 0x7fc2ec0703f0 2023-12-06T03:52:08.213 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.209+0000 7fc2da7fc700 1 -- 172.21.15.132:0/3880157511 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fc2e8025ae0 con 0x7fc2ec0703f0 2023-12-06T03:52:08.213 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.209+0000 7fc2f3e29700 1 -- 172.21.15.132:0/3880157511 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ec045ad0 con 0x7fc2ec0703f0 2023-12-06T03:52:08.213 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.212+0000 7fc2da7fc700 1 -- 172.21.15.132:0/3880157511 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fc2e8016070 con 0x7fc2ec0703f0 2023-12-06T03:52:08.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.280+0000 7fc2f3e29700 1 -- 172.21.15.132:0/3880157511 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc2ec0ffd20 con 0x7fc2ec0703f0 2023-12-06T03:52:08.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.281+0000 7fc2da7fc700 1 -- 172.21.15.132:0/3880157511 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7fc2e801bdc0 con 0x7fc2ec0703f0 2023-12-06T03:52:08.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: { 2023-12-06T03:52:08.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "fsid": "c121c6d8-93ea-11ee-95a2-87774f69a715", 2023-12-06T03:52:08.289 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "health": { 2023-12-06T03:52:08.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-06T03:52:08.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "checks": {}, 2023-12-06T03:52:08.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "mutes": [] 2023-12-06T03:52:08.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:08.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-06T03:52:08.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "quorum": [ 2023-12-06T03:52:08.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 0 2023-12-06T03:52:08.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: ], 2023-12-06T03:52:08.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-06T03:52:08.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "smithi132" 2023-12-06T03:52:08.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: ], 2023-12-06T03:52:08.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-12-06T03:52:08.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "monmap": { 2023-12-06T03:52:08.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "osdmap": { 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "pgmap": { 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-06T03:52:08.291 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "fsmap": { 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "available": false, 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "modules": [ 2023-12-06T03:52:08.292 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "iostat", 2023-12-06T03:52:08.293 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "restful" 2023-12-06T03:52:08.293 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: ], 2023-12-06T03:52:08.293 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "services": {} 2023-12-06T03:52:08.293 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:08.293 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "servicemap": { 2023-12-06T03:52:08.293 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-06T03:52:08.293 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "modified": "2023-12-06T03:52:03.327079+0000", 2023-12-06T03:52:08.293 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "services": {} 2023-12-06T03:52:08.293 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:08.293 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-06T03:52:08.293 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: } 2023-12-06T03:52:08.294 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.282+0000 7fc2f3e29700 1 -- 172.21.15.132:0/3880157511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec0703f0 msgr2=0x7fc2ec10aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:08.294 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.282+0000 7fc2f3e29700 1 --2- 172.21.15.132:0/3880157511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec0703f0 0x7fc2ec10aa50 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fc2e800e040 tx=0x7fc2e8006ee0).stop 2023-12-06T03:52:08.294 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.282+0000 7fc2f3e29700 1 -- 172.21.15.132:0/3880157511 shutdown_connections 2023-12-06T03:52:08.294 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.282+0000 7fc2f3e29700 1 --2- 172.21.15.132:0/3880157511 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec0703f0 0x7fc2ec10aa50 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:08.294 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.282+0000 7fc2f3e29700 1 -- 172.21.15.132:0/3880157511 >> 172.21.15.132:0/3880157511 conn(0x7fc2ec0f63c0 msgr2=0x7fc2ec0f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:08.294 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.282+0000 7fc2f3e29700 1 -- 172.21.15.132:0/3880157511 shutdown_connections 2023-12-06T03:52:08.294 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:08.282+0000 7fc2f3e29700 1 -- 172.21.15.132:0/3880157511 wait complete. 2023-12-06T03:52:08.692 INFO:teuthology.orchestra.run.smithi132.stderr:mgr not available, waiting (1/15)... 2023-12-06T03:52:09.356 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:08 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:08.282470+0000 mon.smithi132 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.132:0/3880157511' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-06T03:52:10.695 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-06T03:52:11.065 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.062+0000 7f87f8885700 1 Processor -- start 2023-12-06T03:52:11.065 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.062+0000 7f87f8885700 1 -- start start 2023-12-06T03:52:11.065 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.062+0000 7f87f8885700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87f4070430 0x7f87f40fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:11.066 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.063+0000 7f87f8885700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40fedd0 con 0x7f87f4070430 2023-12-06T03:52:11.066 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.063+0000 7f87f259c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87f4070430 0x7f87f40fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:11.066 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.063+0000 7f87f259c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87f4070430 0x7f87f40fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49874/0 (socket says 172.21.15.132:49874) 2023-12-06T03:52:11.066 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.063+0000 7f87f259c700 1 -- 172.21.15.132:0/340986652 learned_addr learned my addr 172.21.15.132:0/340986652 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:11.066 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.063+0000 7f87f259c700 1 -- 172.21.15.132:0/340986652 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87f40fef10 con 0x7f87f4070430 2023-12-06T03:52:11.066 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.063+0000 7f87f259c700 1 --2- 172.21.15.132:0/340986652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87f4070430 0x7f87f40fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f87e400b5f0 tx=0x7f87e4006ee0).ready entity=mon.0 client_cookie=d45b2fff8bef08ab server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:11.066 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.063+0000 7f87f159a700 1 -- 172.21.15.132:0/340986652 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87e400d600 con 0x7f87f4070430 2023-12-06T03:52:11.066 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.063+0000 7f87f159a700 1 -- 172.21.15.132:0/340986652 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f87e400dbd0 con 0x7f87f4070430 2023-12-06T03:52:11.066 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.063+0000 7f87f159a700 1 -- 172.21.15.132:0/340986652 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87e4011c00 con 0x7f87f4070430 2023-12-06T03:52:11.068 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.064+0000 7f87f8885700 1 -- 172.21.15.132:0/340986652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87f4070430 msgr2=0x7f87f40fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:11.069 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.064+0000 7f87f8885700 1 --2- 172.21.15.132:0/340986652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87f4070430 0x7f87f40fe8e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f87e400b5f0 tx=0x7f87e4006ee0).stop 2023-12-06T03:52:11.069 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.064+0000 7f87f8885700 1 -- 172.21.15.132:0/340986652 shutdown_connections 2023-12-06T03:52:11.069 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.064+0000 7f87f8885700 1 --2- 172.21.15.132:0/340986652 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87f4070430 0x7f87f40fe8e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:11.069 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.064+0000 7f87f8885700 1 -- 172.21.15.132:0/340986652 >> 172.21.15.132:0/340986652 conn(0x7f87f40f63c0 msgr2=0x7f87f40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:11.069 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.064+0000 7f87f8885700 1 -- 172.21.15.132:0/340986652 shutdown_connections 2023-12-06T03:52:11.069 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.064+0000 7f87f8885700 1 -- 172.21.15.132:0/340986652 wait complete. 2023-12-06T03:52:11.069 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.065+0000 7f87f8885700 1 Processor -- start 2023-12-06T03:52:11.069 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.065+0000 7f87f8885700 1 -- start start 2023-12-06T03:52:11.069 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.065+0000 7f87f8885700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87f4070430 0x7f87f40fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:11.069 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.065+0000 7f87f8885700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87f40fa7e0 con 0x7f87f4070430 2023-12-06T03:52:11.069 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.065+0000 7f87f259c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87f4070430 0x7f87f40fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:11.069 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.065+0000 7f87f259c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87f4070430 0x7f87f40fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49878/0 (socket says 172.21.15.132:49878) 2023-12-06T03:52:11.069 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.065+0000 7f87f259c700 1 -- 172.21.15.132:0/1105804308 learned_addr learned my addr 172.21.15.132:0/1105804308 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:11.070 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.065+0000 7f87f259c700 1 -- 172.21.15.132:0/1105804308 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87e400b200 con 0x7f87f4070430 2023-12-06T03:52:11.070 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.065+0000 7f87f259c700 1 --2- 172.21.15.132:0/1105804308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87f4070430 0x7f87f40fc360 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f87e400b190 tx=0x7f87e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:11.070 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.065+0000 7f87e2ffd700 1 -- 172.21.15.132:0/1105804308 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87e401b410 con 0x7f87f4070430 2023-12-06T03:52:11.070 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.066+0000 7f87e2ffd700 1 -- 172.21.15.132:0/1105804308 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f87e401b9e0 con 0x7f87f4070430 2023-12-06T03:52:11.070 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.066+0000 7f87f8885700 1 -- 172.21.15.132:0/1105804308 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87f40fa980 con 0x7f87f4070430 2023-12-06T03:52:11.070 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.066+0000 7f87e2ffd700 1 -- 172.21.15.132:0/1105804308 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87e401a880 con 0x7f87f4070430 2023-12-06T03:52:11.070 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.066+0000 7f87f8885700 1 -- 172.21.15.132:0/1105804308 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87f40fad90 con 0x7f87f4070430 2023-12-06T03:52:11.070 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.066+0000 7f87e2ffd700 1 -- 172.21.15.132:0/1105804308 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f87e400f070 con 0x7f87f4070430 2023-12-06T03:52:11.070 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.066+0000 7f87e2ffd700 1 -- 172.21.15.132:0/1105804308 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f87e4025b40 con 0x7f87f4070430 2023-12-06T03:52:11.070 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.067+0000 7f87f8885700 1 -- 172.21.15.132:0/1105804308 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87f4045ad0 con 0x7f87f4070430 2023-12-06T03:52:11.071 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.069+0000 7f87e2ffd700 1 -- 172.21.15.132:0/1105804308 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f87e4016070 con 0x7f87f4070430 2023-12-06T03:52:11.141 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.138+0000 7f87f8885700 1 -- 172.21.15.132:0/1105804308 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f87f4104820 con 0x7f87f4070430 2023-12-06T03:52:11.141 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.138+0000 7f87e2ffd700 1 -- 172.21.15.132:0/1105804308 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f87e4025d80 con 0x7f87f4070430 2023-12-06T03:52:11.141 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.141 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: { 2023-12-06T03:52:11.141 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "fsid": "c121c6d8-93ea-11ee-95a2-87774f69a715", 2023-12-06T03:52:11.141 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "health": { 2023-12-06T03:52:11.141 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-06T03:52:11.141 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "checks": {}, 2023-12-06T03:52:11.142 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "mutes": [] 2023-12-06T03:52:11.142 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:11.142 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-06T03:52:11.142 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "quorum": [ 2023-12-06T03:52:11.142 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 0 2023-12-06T03:52:11.144 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: ], 2023-12-06T03:52:11.144 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-06T03:52:11.144 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "smithi132" 2023-12-06T03:52:11.144 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: ], 2023-12-06T03:52:11.144 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-12-06T03:52:11.144 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "monmap": { 2023-12-06T03:52:11.144 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-06T03:52:11.144 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-06T03:52:11.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-06T03:52:11.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:11.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "osdmap": { 2023-12-06T03:52:11.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-06T03:52:11.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-06T03:52:11.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-06T03:52:11.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-06T03:52:11.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-06T03:52:11.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-06T03:52:11.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-06T03:52:11.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:11.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "pgmap": { 2023-12-06T03:52:11.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-06T03:52:11.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-06T03:52:11.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-06T03:52:11.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-06T03:52:11.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-06T03:52:11.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-06T03:52:11.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-06T03:52:11.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-06T03:52:11.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:11.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "fsmap": { 2023-12-06T03:52:11.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-06T03:52:11.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-06T03:52:11.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-06T03:52:11.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:11.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "available": false, 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "modules": [ 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "iostat", 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "restful" 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: ], 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "services": {} 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "servicemap": { 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "modified": "2023-12-06T03:52:03.327079+0000", 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "services": {} 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-06T03:52:11.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: } 2023-12-06T03:52:11.148 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.139+0000 7f87f8885700 1 -- 172.21.15.132:0/1105804308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87f4070430 msgr2=0x7f87f40fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:11.148 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.139+0000 7f87f8885700 1 --2- 172.21.15.132:0/1105804308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87f4070430 0x7f87f40fc360 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f87e400b190 tx=0x7f87e4006ee0).stop 2023-12-06T03:52:11.148 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.139+0000 7f87f8885700 1 -- 172.21.15.132:0/1105804308 shutdown_connections 2023-12-06T03:52:11.148 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.139+0000 7f87f8885700 1 --2- 172.21.15.132:0/1105804308 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87f4070430 0x7f87f40fc360 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:11.148 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.139+0000 7f87f8885700 1 -- 172.21.15.132:0/1105804308 >> 172.21.15.132:0/1105804308 conn(0x7f87f40f63c0 msgr2=0x7f87f40f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:11.148 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.142+0000 7f87f8885700 1 -- 172.21.15.132:0/1105804308 shutdown_connections 2023-12-06T03:52:11.148 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:11.142+0000 7f87f8885700 1 -- 172.21.15.132:0/1105804308 wait complete. 2023-12-06T03:52:11.427 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:11 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:11.139868+0000 mon.smithi132 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.132:0/1105804308' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-06T03:52:11.499 INFO:teuthology.orchestra.run.smithi132.stderr:mgr not available, waiting (2/15)... 2023-12-06T03:52:13.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:12.740548+0000 mon.smithi132 (mon.0) 9 : cluster [INF] Activating manager daemon smithi132.gokxja 2023-12-06T03:52:13.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:12.743447+0000 mon.smithi132 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi132.gokxja(active, starting, since 0.00302846s) 2023-12-06T03:52:13.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:12.744284+0000 mon.smithi132 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.132:0/1748744008' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-06T03:52:13.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:12.744410+0000 mon.smithi132 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.132:0/1748744008' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-06T03:52:13.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:12.744608+0000 mon.smithi132 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.132:0/1748744008' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-06T03:52:13.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:12.744786+0000 mon.smithi132 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.132:0/1748744008' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mon metadata", "id": "smithi132"}]: dispatch 2023-12-06T03:52:13.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:12.745045+0000 mon.smithi132 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.132:0/1748744008' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mgr metadata", "who": "smithi132.gokxja", "id": "smithi132.gokxja"}]: dispatch 2023-12-06T03:52:13.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:12.758938+0000 mon.smithi132 (mon.0) 16 : cluster [INF] Manager daemon smithi132.gokxja is now available 2023-12-06T03:52:13.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:12.767601+0000 mon.smithi132 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.132:0/1748744008' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:13.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:12.770602+0000 mon.smithi132 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.132:0/1748744008' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/mirror_snapshot_schedule"}]: dispatch 2023-12-06T03:52:13.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:12.771654+0000 mon.smithi132 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.132:0/1748744008' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/trash_purge_schedule"}]: dispatch 2023-12-06T03:52:13.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:12.775438+0000 mon.smithi132 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.132:0/1748744008' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:13.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:12.777370+0000 mon.smithi132 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.132:0/1748744008' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:13.502 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-12-06T03:52:13.853 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.849+0000 7fd4290b1700 1 Processor -- start 2023-12-06T03:52:13.853 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.850+0000 7fd4290b1700 1 -- start start 2023-12-06T03:52:13.854 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.850+0000 7fd4290b1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4240703b0 0x7fd424100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:13.854 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.850+0000 7fd4290b1700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4240fc750 con 0x7fd4240703b0 2023-12-06T03:52:13.854 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.850+0000 7fd422d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4240703b0 0x7fd424100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:13.854 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.850+0000 7fd422d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4240703b0 0x7fd424100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49930/0 (socket says 172.21.15.132:49930) 2023-12-06T03:52:13.854 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.850+0000 7fd422d9d700 1 -- 172.21.15.132:0/2754522806 learned_addr learned my addr 172.21.15.132:0/2754522806 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:13.854 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.850+0000 7fd422d9d700 1 -- 172.21.15.132:0/2754522806 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd424101250 con 0x7fd4240703b0 2023-12-06T03:52:13.854 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.850+0000 7fd422d9d700 1 --2- 172.21.15.132:0/2754522806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4240703b0 0x7fd424100d60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd41400b5f0 tx=0x7fd414006ee0).ready entity=mon.0 client_cookie=604e0823c4bb367 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:13.854 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.850+0000 7fd421d9b700 1 -- 172.21.15.132:0/2754522806 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd41400d600 con 0x7fd4240703b0 2023-12-06T03:52:13.854 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.851+0000 7fd421d9b700 1 -- 172.21.15.132:0/2754522806 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd41400dbd0 con 0x7fd4240703b0 2023-12-06T03:52:13.854 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.851+0000 7fd421d9b700 1 -- 172.21.15.132:0/2754522806 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd414011c00 con 0x7fd4240703b0 2023-12-06T03:52:13.854 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.851+0000 7fd4290b1700 1 -- 172.21.15.132:0/2754522806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4240703b0 msgr2=0x7fd424100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:13.854 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.851+0000 7fd4290b1700 1 --2- 172.21.15.132:0/2754522806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4240703b0 0x7fd424100d60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd41400b5f0 tx=0x7fd414006ee0).stop 2023-12-06T03:52:13.854 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.851+0000 7fd4290b1700 1 -- 172.21.15.132:0/2754522806 shutdown_connections 2023-12-06T03:52:13.855 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.851+0000 7fd4290b1700 1 --2- 172.21.15.132:0/2754522806 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4240703b0 0x7fd424100d60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:13.855 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.851+0000 7fd4290b1700 1 -- 172.21.15.132:0/2754522806 >> 172.21.15.132:0/2754522806 conn(0x7fd4240f63c0 msgr2=0x7fd4240f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:13.855 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.851+0000 7fd4290b1700 1 -- 172.21.15.132:0/2754522806 shutdown_connections 2023-12-06T03:52:13.855 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.852+0000 7fd4290b1700 1 -- 172.21.15.132:0/2754522806 wait complete. 2023-12-06T03:52:13.856 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.852+0000 7fd4290b1700 1 Processor -- start 2023-12-06T03:52:13.856 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.852+0000 7fd4290b1700 1 -- start start 2023-12-06T03:52:13.856 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.853+0000 7fd4290b1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4240703b0 0x7fd42410ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.853+0000 7fd4290b1700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd42410d1e0 con 0x7fd4240703b0 2023-12-06T03:52:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.853+0000 7fd422d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4240703b0 0x7fd42410ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.853+0000 7fd422d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4240703b0 0x7fd42410ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49932/0 (socket says 172.21.15.132:49932) 2023-12-06T03:52:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.853+0000 7fd422d9d700 1 -- 172.21.15.132:0/3742468062 learned_addr learned my addr 172.21.15.132:0/3742468062 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.853+0000 7fd422d9d700 1 -- 172.21.15.132:0/3742468062 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd41400b200 con 0x7fd4240703b0 2023-12-06T03:52:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.853+0000 7fd422d9d700 1 --2- 172.21.15.132:0/3742468062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4240703b0 0x7fd42410ccf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fd41400e040 tx=0x7fd414006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.853+0000 7fd41b7fe700 1 -- 172.21.15.132:0/3742468062 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd41400d9f0 con 0x7fd4240703b0 2023-12-06T03:52:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.853+0000 7fd41b7fe700 1 -- 172.21.15.132:0/3742468062 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd41401a9f0 con 0x7fd4240703b0 2023-12-06T03:52:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.853+0000 7fd41b7fe700 1 -- 172.21.15.132:0/3742468062 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd41401b8c0 con 0x7fd4240703b0 2023-12-06T03:52:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.853+0000 7fd4290b1700 1 -- 172.21.15.132:0/3742468062 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd42410d3e0 con 0x7fd4240703b0 2023-12-06T03:52:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.853+0000 7fd4290b1700 1 -- 172.21.15.132:0/3742468062 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd42410d7f0 con 0x7fd4240703b0 2023-12-06T03:52:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.854+0000 7fd41b7fe700 1 -- 172.21.15.132:0/3742468062 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7fd41400f070 con 0x7fd4240703b0 2023-12-06T03:52:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.854+0000 7fd41b7fe700 1 --2- 172.21.15.132:0/3742468062 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7fd40402fcc0 0x7fd404032180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:13.857 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.854+0000 7fd4290b1700 1 -- 172.21.15.132:0/3742468062 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd424045ad0 con 0x7fd4240703b0 2023-12-06T03:52:13.858 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.854+0000 7fd41b7fe700 1 -- 172.21.15.132:0/3742468062 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fd414041070 con 0x7fd4240703b0 2023-12-06T03:52:13.858 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.854+0000 7fd42259c700 1 --2- 172.21.15.132:0/3742468062 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7fd40402fcc0 0x7fd404032180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:13.858 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.855+0000 7fd42259c700 1 --2- 172.21.15.132:0/3742468062 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7fd40402fcc0 0x7fd404032180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd40c0096a0 tx=0x7fd40c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:13.860 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:13.859+0000 7fd41b7fe700 1 -- 172.21.15.132:0/3742468062 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd414016070 con 0x7fd4240703b0 2023-12-06T03:52:14.071 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.061+0000 7fd4290b1700 1 -- 172.21.15.132:0/3742468062 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fd424101db0 con 0x7fd4240703b0 2023-12-06T03:52:14.071 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.061+0000 7fd41b7fe700 1 -- 172.21.15.132:0/3742468062 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7fd414019ce0 con 0x7fd4240703b0 2023-12-06T03:52:14.071 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.071 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: { 2023-12-06T03:52:14.071 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "fsid": "c121c6d8-93ea-11ee-95a2-87774f69a715", 2023-12-06T03:52:14.071 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "health": { 2023-12-06T03:52:14.071 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-06T03:52:14.071 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "checks": {}, 2023-12-06T03:52:14.071 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "mutes": [] 2023-12-06T03:52:14.072 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:14.072 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-06T03:52:14.072 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "quorum": [ 2023-12-06T03:52:14.072 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 0 2023-12-06T03:52:14.072 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: ], 2023-12-06T03:52:14.072 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-06T03:52:14.072 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "smithi132" 2023-12-06T03:52:14.072 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: ], 2023-12-06T03:52:14.072 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-12-06T03:52:14.072 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "monmap": { 2023-12-06T03:52:14.072 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-06T03:52:14.072 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-06T03:52:14.072 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-06T03:52:14.073 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:14.073 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "osdmap": { 2023-12-06T03:52:14.073 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-06T03:52:14.073 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-06T03:52:14.073 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-06T03:52:14.073 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-06T03:52:14.073 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-06T03:52:14.073 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-06T03:52:14.073 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-06T03:52:14.073 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:14.073 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "pgmap": { 2023-12-06T03:52:14.073 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-06T03:52:14.073 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-06T03:52:14.074 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-06T03:52:14.074 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-06T03:52:14.074 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-06T03:52:14.074 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-06T03:52:14.074 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-06T03:52:14.074 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-06T03:52:14.074 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:14.074 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "fsmap": { 2023-12-06T03:52:14.074 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-06T03:52:14.074 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-06T03:52:14.074 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-06T03:52:14.074 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:14.075 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-06T03:52:14.075 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "available": true, 2023-12-06T03:52:14.075 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-06T03:52:14.075 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "modules": [ 2023-12-06T03:52:14.075 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "iostat", 2023-12-06T03:52:14.075 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "restful" 2023-12-06T03:52:14.075 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: ], 2023-12-06T03:52:14.075 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "services": {} 2023-12-06T03:52:14.075 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:14.075 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "servicemap": { 2023-12-06T03:52:14.075 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-06T03:52:14.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "modified": "2023-12-06T03:52:03.327079+0000", 2023-12-06T03:52:14.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "services": {} 2023-12-06T03:52:14.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: }, 2023-12-06T03:52:14.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-06T03:52:14.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: } 2023-12-06T03:52:14.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.064+0000 7fd4290b1700 1 -- 172.21.15.132:0/3742468062 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7fd40402fcc0 msgr2=0x7fd404032180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:14.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.064+0000 7fd4290b1700 1 --2- 172.21.15.132:0/3742468062 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7fd40402fcc0 0x7fd404032180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fd40c0096a0 tx=0x7fd40c006b40).stop 2023-12-06T03:52:14.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.064+0000 7fd4290b1700 1 -- 172.21.15.132:0/3742468062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4240703b0 msgr2=0x7fd42410ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:14.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.064+0000 7fd4290b1700 1 --2- 172.21.15.132:0/3742468062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4240703b0 0x7fd42410ccf0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fd41400e040 tx=0x7fd414006ee0).stop 2023-12-06T03:52:14.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.064+0000 7fd4290b1700 1 -- 172.21.15.132:0/3742468062 shutdown_connections 2023-12-06T03:52:14.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.064+0000 7fd4290b1700 1 --2- 172.21.15.132:0/3742468062 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7fd40402fcc0 0x7fd404032180 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:14.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.064+0000 7fd4290b1700 1 --2- 172.21.15.132:0/3742468062 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd4240703b0 0x7fd42410ccf0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:14.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.064+0000 7fd4290b1700 1 -- 172.21.15.132:0/3742468062 >> 172.21.15.132:0/3742468062 conn(0x7fd4240f63c0 msgr2=0x7fd4240f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:14.077 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.064+0000 7fd4290b1700 1 -- 172.21.15.132:0/3742468062 shutdown_connections 2023-12-06T03:52:14.077 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.064+0000 7fd4290b1700 1 -- 172.21.15.132:0/3742468062 wait complete. 2023-12-06T03:52:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:mgr is available 2023-12-06T03:52:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:Enabling cephadm module... 2023-12-06T03:52:14.416 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-12-06T03:52:15.107 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.975+0000 7f6863329700 1 Processor -- start 2023-12-06T03:52:15.107 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.975+0000 7f6863329700 1 -- start start 2023-12-06T03:52:15.107 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.975+0000 7f6863329700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685c0703d0 0x7f685c100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:15.107 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.975+0000 7f6863329700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685c0fc730 con 0x7f685c0703d0 2023-12-06T03:52:15.107 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.975+0000 7f68610c5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685c0703d0 0x7f685c100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:15.107 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.976+0000 7f68610c5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685c0703d0 0x7f685c100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49940/0 (socket says 172.21.15.132:49940) 2023-12-06T03:52:15.107 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.976+0000 7f68610c5700 1 -- 172.21.15.132:0/340694851 learned_addr learned my addr 172.21.15.132:0/340694851 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:15.107 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.976+0000 7f68610c5700 1 -- 172.21.15.132:0/340694851 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685c101230 con 0x7f685c0703d0 2023-12-06T03:52:15.108 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.976+0000 7f68610c5700 1 --2- 172.21.15.132:0/340694851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685c0703d0 0x7f685c100d40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f684c00b5f0 tx=0x7f684c006ee0).ready entity=mon.0 client_cookie=9d5b98bd5ba58c3b server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:15.108 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.976+0000 7f684bfff700 1 -- 172.21.15.132:0/340694851 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f684c00d600 con 0x7f685c0703d0 2023-12-06T03:52:15.108 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.976+0000 7f684bfff700 1 -- 172.21.15.132:0/340694851 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f684c00dbd0 con 0x7f685c0703d0 2023-12-06T03:52:15.108 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.976+0000 7f684bfff700 1 -- 172.21.15.132:0/340694851 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f684c011c00 con 0x7f685c0703d0 2023-12-06T03:52:15.108 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.976+0000 7f6863329700 1 -- 172.21.15.132:0/340694851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685c0703d0 msgr2=0x7f685c100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:15.108 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.976+0000 7f6863329700 1 --2- 172.21.15.132:0/340694851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685c0703d0 0x7f685c100d40 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f684c00b5f0 tx=0x7f684c006ee0).stop 2023-12-06T03:52:15.108 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.976+0000 7f6863329700 1 -- 172.21.15.132:0/340694851 shutdown_connections 2023-12-06T03:52:15.108 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.976+0000 7f6863329700 1 --2- 172.21.15.132:0/340694851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685c0703d0 0x7f685c100d40 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:15.108 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.976+0000 7f6863329700 1 -- 172.21.15.132:0/340694851 >> 172.21.15.132:0/340694851 conn(0x7f685c0f63c0 msgr2=0x7f685c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:15.108 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.977+0000 7f6863329700 1 -- 172.21.15.132:0/340694851 shutdown_connections 2023-12-06T03:52:15.108 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.977+0000 7f6863329700 1 -- 172.21.15.132:0/340694851 wait complete. 2023-12-06T03:52:15.108 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.977+0000 7f6863329700 1 Processor -- start 2023-12-06T03:52:15.108 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.977+0000 7f6863329700 1 -- start start 2023-12-06T03:52:15.108 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.977+0000 7f6863329700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685c0703d0 0x7f685c10cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:15.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:14 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:13.745771+0000 mon.smithi132 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi132.gokxja(active, since 1.00535s) 2023-12-06T03:52:15.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:14 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:14.062998+0000 mon.smithi132 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.132:0/3742468062' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-06T03:52:15.111 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.977+0000 7f6863329700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685c10d0a0 con 0x7f685c0703d0 2023-12-06T03:52:15.111 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.977+0000 7f68610c5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685c0703d0 0x7f685c10cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:15.111 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.977+0000 7f68610c5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685c0703d0 0x7f685c10cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49950/0 (socket says 172.21.15.132:49950) 2023-12-06T03:52:15.111 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.977+0000 7f68610c5700 1 -- 172.21.15.132:0/2418951153 learned_addr learned my addr 172.21.15.132:0/2418951153 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:15.111 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.978+0000 7f68610c5700 1 -- 172.21.15.132:0/2418951153 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f684c00b200 con 0x7f685c0703d0 2023-12-06T03:52:15.111 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.978+0000 7f68610c5700 1 --2- 172.21.15.132:0/2418951153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685c0703d0 0x7f685c10cbb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f684c00b190 tx=0x7f684c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:15.111 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.978+0000 7f6849ffb700 1 -- 172.21.15.132:0/2418951153 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f684c01b410 con 0x7f685c0703d0 2023-12-06T03:52:15.111 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.978+0000 7f6849ffb700 1 -- 172.21.15.132:0/2418951153 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f684c01b9e0 con 0x7f685c0703d0 2023-12-06T03:52:15.111 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.978+0000 7f6849ffb700 1 -- 172.21.15.132:0/2418951153 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f684c01a880 con 0x7f685c0703d0 2023-12-06T03:52:15.111 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.978+0000 7f6863329700 1 -- 172.21.15.132:0/2418951153 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685c10d2a0 con 0x7f685c0703d0 2023-12-06T03:52:15.111 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.978+0000 7f6863329700 1 -- 172.21.15.132:0/2418951153 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f685c10d6b0 con 0x7f685c0703d0 2023-12-06T03:52:15.112 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.979+0000 7f6849ffb700 1 -- 172.21.15.132:0/2418951153 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f684c00f070 con 0x7f685c0703d0 2023-12-06T03:52:15.112 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.979+0000 7f6849ffb700 1 --2- 172.21.15.132:0/2418951153 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f684402fca0 0x7f6844032160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:15.112 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.979+0000 7f6849ffb700 1 -- 172.21.15.132:0/2418951153 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f684c040fe0 con 0x7f685c0703d0 2023-12-06T03:52:15.112 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.979+0000 7f68608c4700 1 --2- 172.21.15.132:0/2418951153 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f684402fca0 0x7f6844032160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:15.112 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.979+0000 7f6863329700 1 -- 172.21.15.132:0/2418951153 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685c045ad0 con 0x7f685c0703d0 2023-12-06T03:52:15.112 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.979+0000 7f68608c4700 1 --2- 172.21.15.132:0/2418951153 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f684402fca0 0x7f6844032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f68580096a0 tx=0x7f6858006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:15.112 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:14.983+0000 7f6849ffb700 1 -- 172.21.15.132:0/2418951153 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f684c016020 con 0x7f685c0703d0 2023-12-06T03:52:15.175 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:15.173+0000 7f6863329700 1 -- 172.21.15.132:0/2418951153 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f685c106a90 con 0x7f685c0703d0 2023-12-06T03:52:15.746 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:15.743+0000 7f6849ffb700 1 -- 172.21.15.132:0/2418951153 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f684c01b570 con 0x7f685c0703d0 2023-12-06T03:52:15.747 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:15.745+0000 7f6849ffb700 1 -- 172.21.15.132:0/2418951153 <== mon.0 v2:172.21.15.132:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v4) v1 ==== 86+0+0 (secure 0 0 0) 0x7f684c023b40 con 0x7f685c0703d0 2023-12-06T03:52:15.751 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:15.748+0000 7f6863329700 1 -- 172.21.15.132:0/2418951153 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f684402fca0 msgr2=0x7f6844032160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:15.751 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:15.748+0000 7f6863329700 1 --2- 172.21.15.132:0/2418951153 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f684402fca0 0x7f6844032160 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f68580096a0 tx=0x7f6858006b40).stop 2023-12-06T03:52:15.751 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:15.748+0000 7f6863329700 1 -- 172.21.15.132:0/2418951153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685c0703d0 msgr2=0x7f685c10cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:15.751 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:15.748+0000 7f6863329700 1 --2- 172.21.15.132:0/2418951153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685c0703d0 0x7f685c10cbb0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f684c00b190 tx=0x7f684c006ee0).stop 2023-12-06T03:52:15.751 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:15.748+0000 7f6863329700 1 -- 172.21.15.132:0/2418951153 shutdown_connections 2023-12-06T03:52:15.751 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:15.748+0000 7f6863329700 1 --2- 172.21.15.132:0/2418951153 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f684402fca0 0x7f6844032160 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:15.751 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:15.748+0000 7f6863329700 1 --2- 172.21.15.132:0/2418951153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f685c0703d0 0x7f685c10cbb0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:15.752 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:15.748+0000 7f6863329700 1 -- 172.21.15.132:0/2418951153 >> 172.21.15.132:0/2418951153 conn(0x7f685c0f63c0 msgr2=0x7f685c0f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:15.752 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:15.748+0000 7f6863329700 1 -- 172.21.15.132:0/2418951153 shutdown_connections 2023-12-06T03:52:15.752 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:15.748+0000 7f6863329700 1 -- 172.21.15.132:0/2418951153 wait complete. 2023-12-06T03:52:16.035 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:15 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:15.175245+0000 mon.smithi132 (mon.0) 24 : audit [INF] from='client.? 172.21.15.132:0/2418951153' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-12-06T03:52:16.116 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-12-06T03:52:16.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.459+0000 7f074d64a700 1 Processor -- start 2023-12-06T03:52:16.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.459+0000 7f074d64a700 1 -- start start 2023-12-06T03:52:16.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.460+0000 7f074d64a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07480fab80 0x7f07480fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:16.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.460+0000 7f074d64a700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07480fb490 con 0x7f07480fab80 2023-12-06T03:52:16.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.460+0000 7f0746ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07480fab80 0x7f07480fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:16.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.460+0000 7f0746ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07480fab80 0x7f07480fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49966/0 (socket says 172.21.15.132:49966) 2023-12-06T03:52:16.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.460+0000 7f0746ffd700 1 -- 172.21.15.132:0/3614906068 learned_addr learned my addr 172.21.15.132:0/3614906068 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:16.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.460+0000 7f0746ffd700 1 -- 172.21.15.132:0/3614906068 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07480fb5d0 con 0x7f07480fab80 2023-12-06T03:52:16.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.460+0000 7f0746ffd700 1 --2- 172.21.15.132:0/3614906068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07480fab80 0x7f07480fafa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f073800b5f0 tx=0x7f0738006ee0).ready entity=mon.0 client_cookie=9c1a6937237c3c36 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:16.465 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.461+0000 7f0745ffb700 1 -- 172.21.15.132:0/3614906068 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f073800d600 con 0x7f07480fab80 2023-12-06T03:52:16.465 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.461+0000 7f0745ffb700 1 -- 172.21.15.132:0/3614906068 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f073800dbd0 con 0x7f07480fab80 2023-12-06T03:52:16.465 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.461+0000 7f0745ffb700 1 -- 172.21.15.132:0/3614906068 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0738011c00 con 0x7f07480fab80 2023-12-06T03:52:16.465 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.461+0000 7f074d64a700 1 -- 172.21.15.132:0/3614906068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07480fab80 msgr2=0x7f07480fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:16.465 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.461+0000 7f074d64a700 1 --2- 172.21.15.132:0/3614906068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07480fab80 0x7f07480fafa0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f073800b5f0 tx=0x7f0738006ee0).stop 2023-12-06T03:52:16.465 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.461+0000 7f074d64a700 1 -- 172.21.15.132:0/3614906068 shutdown_connections 2023-12-06T03:52:16.465 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.461+0000 7f074d64a700 1 --2- 172.21.15.132:0/3614906068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07480fab80 0x7f07480fafa0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:16.465 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.461+0000 7f074d64a700 1 -- 172.21.15.132:0/3614906068 >> 172.21.15.132:0/3614906068 conn(0x7f07480f63c0 msgr2=0x7f07480f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:16.465 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.461+0000 7f074d64a700 1 -- 172.21.15.132:0/3614906068 shutdown_connections 2023-12-06T03:52:16.465 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.461+0000 7f074d64a700 1 -- 172.21.15.132:0/3614906068 wait complete. 2023-12-06T03:52:16.465 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.462+0000 7f074d64a700 1 Processor -- start 2023-12-06T03:52:16.467 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.462+0000 7f074d64a700 1 -- start start 2023-12-06T03:52:16.467 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.462+0000 7f074d64a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07480fab80 0x7f074810d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:16.467 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.462+0000 7f074d64a700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07480704b0 con 0x7f07480fab80 2023-12-06T03:52:16.467 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.463+0000 7f0746ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07480fab80 0x7f074810d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:16.468 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.463+0000 7f0746ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07480fab80 0x7f074810d6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:49980/0 (socket says 172.21.15.132:49980) 2023-12-06T03:52:16.468 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.463+0000 7f0746ffd700 1 -- 172.21.15.132:0/1513353309 learned_addr learned my addr 172.21.15.132:0/1513353309 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:16.468 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.463+0000 7f0746ffd700 1 -- 172.21.15.132:0/1513353309 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f073800b200 con 0x7f07480fab80 2023-12-06T03:52:16.469 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.463+0000 7f0746ffd700 1 --2- 172.21.15.132:0/1513353309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07480fab80 0x7f074810d6c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f073800b1d0 tx=0x7f0738006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:16.469 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.463+0000 7f072f7fe700 1 -- 172.21.15.132:0/1513353309 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f073801b410 con 0x7f07480fab80 2023-12-06T03:52:16.469 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.463+0000 7f074d64a700 1 -- 172.21.15.132:0/1513353309 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07480706b0 con 0x7f07480fab80 2023-12-06T03:52:16.469 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.463+0000 7f072f7fe700 1 -- 172.21.15.132:0/1513353309 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f073801b9e0 con 0x7f07480fab80 2023-12-06T03:52:16.469 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.463+0000 7f072f7fe700 1 -- 172.21.15.132:0/1513353309 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f073801a840 con 0x7f07480fab80 2023-12-06T03:52:16.469 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.463+0000 7f074d64a700 1 -- 172.21.15.132:0/1513353309 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f074810dd70 con 0x7f07480fab80 2023-12-06T03:52:16.469 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.464+0000 7f072f7fe700 1 -- 172.21.15.132:0/1513353309 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f073800f070 con 0x7f07480fab80 2023-12-06T03:52:16.469 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.464+0000 7f074d64a700 1 -- 172.21.15.132:0/1513353309 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0748045ad0 con 0x7f07480fab80 2023-12-06T03:52:16.469 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.464+0000 7f072f7fe700 1 --2- 172.21.15.132:0/1513353309 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f073402fdb0 0x7f0734032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:16.469 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.464+0000 7f072f7fe700 1 -- 172.21.15.132:0/1513353309 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0738041280 con 0x7f07480fab80 2023-12-06T03:52:16.469 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.464+0000 7f07467fc700 1 -- 172.21.15.132:0/1513353309 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f073402fdb0 msgr2=0x7f0734032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/1406682404 2023-12-06T03:52:16.469 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.464+0000 7f07467fc700 1 --2- 172.21.15.132:0/1513353309 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f073402fdb0 0x7f0734032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-06T03:52:16.470 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.469+0000 7f072f7fe700 1 -- 172.21.15.132:0/1513353309 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0738022d70 con 0x7f07480fab80 2023-12-06T03:52:16.666 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.663+0000 7f074d64a700 1 -- 172.21.15.132:0/1513353309 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f07480fc890 con 0x7f07480fab80 2023-12-06T03:52:16.667 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.664+0000 7f072f7fe700 1 -- 172.21.15.132:0/1513353309 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v4) v1 ==== 56+0+103 (secure 0 0 0) 0x7f0738016070 con 0x7f07480fab80 2023-12-06T03:52:16.667 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: { 2023-12-06T03:52:16.667 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "epoch": 4, 2023-12-06T03:52:16.667 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "available": true, 2023-12-06T03:52:16.667 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "active_name": "smithi132.gokxja", 2023-12-06T03:52:16.667 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-06T03:52:16.667 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: } 2023-12-06T03:52:16.667 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.665+0000 7f07467fc700 1 -- 172.21.15.132:0/1513353309 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f073402fdb0 msgr2=0x7f0734032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/1406682404 2023-12-06T03:52:16.667 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.665+0000 7f07467fc700 1 --2- 172.21.15.132:0/1513353309 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f073402fdb0 0x7f0734032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-06T03:52:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.666+0000 7f074d64a700 1 -- 172.21.15.132:0/1513353309 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f073402fdb0 msgr2=0x7f0734032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-06T03:52:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.666+0000 7f074d64a700 1 --2- 172.21.15.132:0/1513353309 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f073402fdb0 0x7f0734032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-06T03:52:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.666+0000 7f074d64a700 1 -- 172.21.15.132:0/1513353309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07480fab80 msgr2=0x7f074810d6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.666+0000 7f074d64a700 1 --2- 172.21.15.132:0/1513353309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07480fab80 0x7f074810d6c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f073800b1d0 tx=0x7f0738006ee0).stop 2023-12-06T03:52:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.666+0000 7f074d64a700 1 -- 172.21.15.132:0/1513353309 shutdown_connections 2023-12-06T03:52:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.666+0000 7f074d64a700 1 --2- 172.21.15.132:0/1513353309 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f073402fdb0 0x7f0734032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-06T03:52:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.666+0000 7f074d64a700 1 --2- 172.21.15.132:0/1513353309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07480fab80 0x7f074810d6c0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.666+0000 7f074d64a700 1 -- 172.21.15.132:0/1513353309 >> 172.21.15.132:0/1513353309 conn(0x7f07480f63c0 msgr2=0x7f07480f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.666+0000 7f074d64a700 1 -- 172.21.15.132:0/1513353309 shutdown_connections 2023-12-06T03:52:16.669 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:16.667+0000 7f074d64a700 1 -- 172.21.15.132:0/1513353309 wait complete. 2023-12-06T03:52:16.968 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:16 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:15.746755+0000 mon.smithi132 (mon.0) 25 : audit [INF] from='client.? 172.21.15.132:0/2418951153' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-12-06T03:52:16.968 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:16 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:15.746819+0000 mon.smithi132 (mon.0) 26 : cluster [DBG] mgrmap e4: smithi132.gokxja(active, since 3s) 2023-12-06T03:52:16.968 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:16 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:16.665614+0000 mon.smithi132 (mon.0) 27 : audit [DBG] from='client.? 172.21.15.132:0/1513353309' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-06T03:52:17.026 INFO:teuthology.orchestra.run.smithi132.stderr:Waiting for the mgr to restart... 2023-12-06T03:52:17.027 INFO:teuthology.orchestra.run.smithi132.stderr:Waiting for mgr epoch 4... 2023-12-06T03:52:17.027 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-12-06T03:52:17.402 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.395+0000 7f1682c8c700 1 Processor -- start 2023-12-06T03:52:17.402 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.396+0000 7f1682c8c700 1 -- start start 2023-12-06T03:52:17.402 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.396+0000 7f1682c8c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f167c070430 0x7f167c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:17.402 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.396+0000 7f1682c8c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c0fedd0 con 0x7f167c070430 2023-12-06T03:52:17.402 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.396+0000 7f1680a28700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f167c070430 0x7f167c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:17.403 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.396+0000 7f1680a28700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f167c070430 0x7f167c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:50038/0 (socket says 172.21.15.132:50038) 2023-12-06T03:52:17.403 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.396+0000 7f1680a28700 1 -- 172.21.15.132:0/289865461 learned_addr learned my addr 172.21.15.132:0/289865461 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:17.403 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.396+0000 7f1680a28700 1 -- 172.21.15.132:0/289865461 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167c0fef10 con 0x7f167c070430 2023-12-06T03:52:17.403 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.396+0000 7f1680a28700 1 --2- 172.21.15.132:0/289865461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f167c070430 0x7f167c0fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f167000b5f0 tx=0x7f1670006ee0).ready entity=mon.0 client_cookie=9b48cd36a410e1ba server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:17.403 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.397+0000 7f167b7fe700 1 -- 172.21.15.132:0/289865461 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167000d600 con 0x7f167c070430 2023-12-06T03:52:17.403 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.397+0000 7f167b7fe700 1 -- 172.21.15.132:0/289865461 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f167000dbd0 con 0x7f167c070430 2023-12-06T03:52:17.403 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.397+0000 7f167b7fe700 1 -- 172.21.15.132:0/289865461 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1670011c00 con 0x7f167c070430 2023-12-06T03:52:17.404 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.397+0000 7f1682c8c700 1 -- 172.21.15.132:0/289865461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f167c070430 msgr2=0x7f167c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:17.404 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.397+0000 7f1682c8c700 1 --2- 172.21.15.132:0/289865461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f167c070430 0x7f167c0fe8e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f167000b5f0 tx=0x7f1670006ee0).stop 2023-12-06T03:52:17.404 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.398+0000 7f1682c8c700 1 -- 172.21.15.132:0/289865461 shutdown_connections 2023-12-06T03:52:17.404 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.398+0000 7f1682c8c700 1 --2- 172.21.15.132:0/289865461 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f167c070430 0x7f167c0fe8e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:17.404 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.398+0000 7f1682c8c700 1 -- 172.21.15.132:0/289865461 >> 172.21.15.132:0/289865461 conn(0x7f167c0f63c0 msgr2=0x7f167c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:17.404 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.398+0000 7f1682c8c700 1 -- 172.21.15.132:0/289865461 shutdown_connections 2023-12-06T03:52:17.404 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.398+0000 7f1682c8c700 1 -- 172.21.15.132:0/289865461 wait complete. 2023-12-06T03:52:17.404 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.398+0000 7f1682c8c700 1 Processor -- start 2023-12-06T03:52:17.404 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.398+0000 7f1682c8c700 1 -- start start 2023-12-06T03:52:17.405 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.398+0000 7f1682c8c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f167c070430 0x7f167c10aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:17.405 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.398+0000 7f1682c8c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f167c10af30 con 0x7f167c070430 2023-12-06T03:52:17.405 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.399+0000 7f1680a28700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f167c070430 0x7f167c10aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:17.405 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.399+0000 7f1680a28700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f167c070430 0x7f167c10aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:50046/0 (socket says 172.21.15.132:50046) 2023-12-06T03:52:17.406 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.399+0000 7f1680a28700 1 -- 172.21.15.132:0/3737605740 learned_addr learned my addr 172.21.15.132:0/3737605740 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:17.406 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.399+0000 7f1680a28700 1 -- 172.21.15.132:0/3737605740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f167000b200 con 0x7f167c070430 2023-12-06T03:52:17.407 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.399+0000 7f1680a28700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f167c070430 0x7f167c10aa40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f167000e040 tx=0x7f1670006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:17.407 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.399+0000 7f16797fa700 1 -- 172.21.15.132:0/3737605740 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167000d9f0 con 0x7f167c070430 2023-12-06T03:52:17.407 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.399+0000 7f1682c8c700 1 -- 172.21.15.132:0/3737605740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f167c10b130 con 0x7f167c070430 2023-12-06T03:52:17.407 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.399+0000 7f16797fa700 1 -- 172.21.15.132:0/3737605740 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f167001a9f0 con 0x7f167c070430 2023-12-06T03:52:17.407 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.399+0000 7f16797fa700 1 -- 172.21.15.132:0/3737605740 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f167001b8c0 con 0x7f167c070430 2023-12-06T03:52:17.407 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.399+0000 7f1682c8c700 1 -- 172.21.15.132:0/3737605740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f167c10b540 con 0x7f167c070430 2023-12-06T03:52:17.407 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.400+0000 7f16797fa700 1 -- 172.21.15.132:0/3737605740 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 4) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f167000f070 con 0x7f167c070430 2023-12-06T03:52:17.408 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.400+0000 7f16797fa700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f166002fd50 0x7f1660032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:17.408 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.400+0000 7f16797fa700 1 -- 172.21.15.132:0/3737605740 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1670042040 con 0x7f167c070430 2023-12-06T03:52:17.408 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.400+0000 7f167bfff700 1 -- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f166002fd50 msgr2=0x7f1660032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/1406682404 2023-12-06T03:52:17.408 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.400+0000 7f167bfff700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f166002fd50 0x7f1660032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-06T03:52:17.408 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.401+0000 7f1682c8c700 1 -- 172.21.15.132:0/3737605740 --> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f167c059100 con 0x7f166002fd50 2023-12-06T03:52:17.603 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.601+0000 7f167bfff700 1 -- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f166002fd50 msgr2=0x7f1660032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/1406682404 2023-12-06T03:52:17.603 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:17.601+0000 7f167bfff700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f166002fd50 0x7f1660032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-06T03:52:18.004 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:18.002+0000 7f167bfff700 1 -- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f166002fd50 msgr2=0x7f1660032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/1406682404 2023-12-06T03:52:18.004 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:18.002+0000 7f167bfff700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f166002fd50 0x7f1660032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-06T03:52:18.805 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:18.803+0000 7f167bfff700 1 -- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f166002fd50 msgr2=0x7f1660032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/1406682404 2023-12-06T03:52:18.805 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:18.803+0000 7f167bfff700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f166002fd50 0x7f1660032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-06T03:52:20.406 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:20.404+0000 7f167bfff700 1 -- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f166002fd50 msgr2=0x7f1660032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/1406682404 2023-12-06T03:52:20.407 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:20.404+0000 7f167bfff700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f166002fd50 0x7f1660032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-06T03:52:20.748 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:20.746+0000 7f16797fa700 1 -- 172.21.15.132:0/3737605740 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mgrmap(e 5) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f16700257e0 con 0x7f167c070430 2023-12-06T03:52:20.748 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:20.746+0000 7f16797fa700 1 -- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f166002fd50 msgr2=0x7f1660032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-06T03:52:20.748 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:20.746+0000 7f16797fa700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f166002fd50 0x7f1660032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-06T03:52:21.752 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.750+0000 7f16797fa700 1 -- 172.21.15.132:0/3737605740 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 6) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f1670042a80 con 0x7f167c070430 2023-12-06T03:52:21.752 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.750+0000 7f16797fa700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f16600336c0 0x7f1660035ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:21.753 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.750+0000 7f16797fa700 1 -- 172.21.15.132:0/3737605740 --> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f167c059100 con 0x7f16600336c0 2023-12-06T03:52:21.753 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.750+0000 7f167bfff700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f16600336c0 0x7f1660035ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:21.753 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.751+0000 7f167bfff700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f16600336c0 0x7f1660035ab0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f1668006fd0 tx=0x7f1668006cf0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:21.753 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.752+0000 7f16797fa700 1 -- 172.21.15.132:0/3737605740 <== mgr.14114 v2:172.21.15.132:6800/3766660979 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f167c059100 con 0x7f16600336c0 2023-12-06T03:52:21.760 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.758+0000 7f1682c8c700 1 -- 172.21.15.132:0/3737605740 --> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f167c045ad0 con 0x7f16600336c0 2023-12-06T03:52:21.760 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.758+0000 7f16797fa700 1 -- 172.21.15.132:0/3737605740 <== mgr.14114 v2:172.21.15.132:6800/3766660979 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f167c045ad0 con 0x7f16600336c0 2023-12-06T03:52:21.760 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: { 2023-12-06T03:52:21.760 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "mgrmap_epoch": 6, 2023-12-06T03:52:21.761 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "initialized": true 2023-12-06T03:52:21.761 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: } 2023-12-06T03:52:21.762 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.758+0000 7f1682c8c700 1 -- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f16600336c0 msgr2=0x7f1660035ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:21.762 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.758+0000 7f1682c8c700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f16600336c0 0x7f1660035ab0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f1668006fd0 tx=0x7f1668006cf0).stop 2023-12-06T03:52:21.762 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.758+0000 7f1682c8c700 1 -- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f167c070430 msgr2=0x7f167c10aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:21.762 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.758+0000 7f1682c8c700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f167c070430 0x7f167c10aa40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f167000e040 tx=0x7f1670006ee0).stop 2023-12-06T03:52:21.762 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.759+0000 7f1682c8c700 1 -- 172.21.15.132:0/3737605740 shutdown_connections 2023-12-06T03:52:21.762 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.759+0000 7f1682c8c700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f16600336c0 0x7f1660035ab0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:21.762 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.759+0000 7f1682c8c700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f167c070430 0x7f167c10aa40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:21.762 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.759+0000 7f1682c8c700 1 --2- 172.21.15.132:0/3737605740 >> [v2:172.21.15.132:6800/1406682404,v1:172.21.15.132:6801/1406682404] conn(0x7f166002fd50 0x7f1660032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-06T03:52:21.763 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.759+0000 7f1682c8c700 1 -- 172.21.15.132:0/3737605740 >> 172.21.15.132:0/3737605740 conn(0x7f167c0f63c0 msgr2=0x7f167c0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:21.763 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.759+0000 7f1682c8c700 1 -- 172.21.15.132:0/3737605740 shutdown_connections 2023-12-06T03:52:21.763 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:21.759+0000 7f1682c8c700 1 -- 172.21.15.132:0/3737605740 wait complete. 2023-12-06T03:52:22.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:20.695297+0000 mon.smithi132 (mon.0) 28 : cluster [INF] Active manager daemon smithi132.gokxja restarted 2023-12-06T03:52:22.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:20.695814+0000 mon.smithi132 (mon.0) 29 : cluster [INF] Activating manager daemon smithi132.gokxja 2023-12-06T03:52:22.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:20.697659+0000 mon.smithi132 (mon.0) 30 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-12-06T03:52:22.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:20.748230+0000 mon.smithi132 (mon.0) 31 : cluster [DBG] mgrmap e5: smithi132.gokxja(active, starting, since 0.0525507s) 2023-12-06T03:52:22.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:20.749340+0000 mon.smithi132 (mon.0) 32 : audit [DBG] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mon metadata", "id": "smithi132"}]: dispatch 2023-12-06T03:52:22.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:20.749879+0000 mon.smithi132 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mgr metadata", "who": "smithi132.gokxja", "id": "smithi132.gokxja"}]: dispatch 2023-12-06T03:52:22.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:20.750523+0000 mon.smithi132 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-06T03:52:22.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:20.750690+0000 mon.smithi132 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-06T03:52:22.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:20.750808+0000 mon.smithi132 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-06T03:52:22.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:20.763713+0000 mon.smithi132 (mon.0) 37 : cluster [INF] Manager daemon smithi132.gokxja is now available 2023-12-06T03:52:22.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:20.779376+0000 mon.smithi132 (mon.0) 38 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:22.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:20.784583+0000 mon.smithi132 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:22.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:20.790081+0000 mon.smithi132 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:22.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:20.881634+0000 mon.smithi132 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:22.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:20.882328+0000 mon.smithi132 (mon.0) 42 : audit [DBG] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-06T03:52:22.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:20.883918+0000 mon.smithi132 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-06T03:52:22.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:20.885398+0000 mon.smithi132 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-06T03:52:22.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:20.894899+0000 mon.smithi132 (mon.0) 45 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/mirror_snapshot_schedule"}]: dispatch 2023-12-06T03:52:22.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:20.896004+0000 mon.smithi132 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/trash_purge_schedule"}]: dispatch 2023-12-06T03:52:22.557 INFO:teuthology.orchestra.run.smithi132.stderr:mgr epoch 4 is available 2023-12-06T03:52:22.558 INFO:teuthology.orchestra.run.smithi132.stderr:Setting orchestrator backend to cephadm... 2023-12-06T03:52:22.558 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-12-06T03:52:22.908 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.903+0000 7ff0fc1e3700 1 Processor -- start 2023-12-06T03:52:22.908 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.903+0000 7ff0fc1e3700 1 -- start start 2023-12-06T03:52:22.908 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.903+0000 7ff0fc1e3700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0f40703b0 0x7ff0f4100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:22.908 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.903+0000 7ff0fc1e3700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f40fc750 con 0x7ff0f40703b0 2023-12-06T03:52:22.908 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.904+0000 7ff0f9f7f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0f40703b0 0x7ff0f4100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:22.908 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.904+0000 7ff0f9f7f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0f40703b0 0x7ff0f4100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:50116/0 (socket says 172.21.15.132:50116) 2023-12-06T03:52:22.908 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.904+0000 7ff0f9f7f700 1 -- 172.21.15.132:0/872047327 learned_addr learned my addr 172.21.15.132:0/872047327 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:22.908 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.904+0000 7ff0f9f7f700 1 -- 172.21.15.132:0/872047327 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f4101250 con 0x7ff0f40703b0 2023-12-06T03:52:22.908 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.904+0000 7ff0f9f7f700 1 --2- 172.21.15.132:0/872047327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0f40703b0 0x7ff0f4100d60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7ff0e400b5f0 tx=0x7ff0e4006ee0).ready entity=mon.0 client_cookie=3e75bf08cecb96f8 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:22.908 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.904+0000 7ff0f8f7d700 1 -- 172.21.15.132:0/872047327 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0e400d600 con 0x7ff0f40703b0 2023-12-06T03:52:22.908 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.904+0000 7ff0f8f7d700 1 -- 172.21.15.132:0/872047327 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff0e400dbd0 con 0x7ff0f40703b0 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.904+0000 7ff0f8f7d700 1 -- 172.21.15.132:0/872047327 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0e4011c00 con 0x7ff0f40703b0 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.905+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/872047327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0f40703b0 msgr2=0x7ff0f4100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.905+0000 7ff0fc1e3700 1 --2- 172.21.15.132:0/872047327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0f40703b0 0x7ff0f4100d60 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7ff0e400b5f0 tx=0x7ff0e4006ee0).stop 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.905+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/872047327 shutdown_connections 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.905+0000 7ff0fc1e3700 1 --2- 172.21.15.132:0/872047327 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0f40703b0 0x7ff0f4100d60 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.905+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/872047327 >> 172.21.15.132:0/872047327 conn(0x7ff0f40f63c0 msgr2=0x7ff0f40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.905+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/872047327 shutdown_connections 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.905+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/872047327 wait complete. 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.905+0000 7ff0fc1e3700 1 Processor -- start 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.905+0000 7ff0fc1e3700 1 -- start start 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.906+0000 7ff0fc1e3700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0f40703b0 0x7ff0f410ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.906+0000 7ff0fc1e3700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f410d1e0 con 0x7ff0f40703b0 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.906+0000 7ff0f9f7f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0f40703b0 0x7ff0f410ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.906+0000 7ff0f9f7f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0f40703b0 0x7ff0f410ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:50120/0 (socket says 172.21.15.132:50120) 2023-12-06T03:52:22.909 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.906+0000 7ff0f9f7f700 1 -- 172.21.15.132:0/4243302822 learned_addr learned my addr 172.21.15.132:0/4243302822 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:22.910 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.906+0000 7ff0f9f7f700 1 -- 172.21.15.132:0/4243302822 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0e400b200 con 0x7ff0f40703b0 2023-12-06T03:52:22.910 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.906+0000 7ff0f9f7f700 1 --2- 172.21.15.132:0/4243302822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0f40703b0 0x7ff0f410ccf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7ff0e400e040 tx=0x7ff0e4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:22.911 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.906+0000 7ff0e27fc700 1 -- 172.21.15.132:0/4243302822 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0e400d9f0 con 0x7ff0f40703b0 2023-12-06T03:52:22.911 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.906+0000 7ff0e27fc700 1 -- 172.21.15.132:0/4243302822 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff0e401a9f0 con 0x7ff0f40703b0 2023-12-06T03:52:22.911 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.906+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/4243302822 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f410d3e0 con 0x7ff0f40703b0 2023-12-06T03:52:22.911 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.906+0000 7ff0e27fc700 1 -- 172.21.15.132:0/4243302822 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0e401b8c0 con 0x7ff0f40703b0 2023-12-06T03:52:22.911 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.906+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/4243302822 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f410d7f0 con 0x7ff0f40703b0 2023-12-06T03:52:22.911 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.907+0000 7ff0e27fc700 1 -- 172.21.15.132:0/4243302822 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff0e400f070 con 0x7ff0f40703b0 2023-12-06T03:52:22.911 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.907+0000 7ff0e27fc700 1 --2- 172.21.15.132:0/4243302822 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7ff0dc02fd50 0x7ff0dc032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:22.911 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.907+0000 7ff0e27fc700 1 -- 172.21.15.132:0/4243302822 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff0e4041320 con 0x7ff0f40703b0 2023-12-06T03:52:22.911 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.907+0000 7ff0f977e700 1 --2- 172.21.15.132:0/4243302822 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7ff0dc02fd50 0x7ff0dc032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:22.911 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.908+0000 7ff0f977e700 1 --2- 172.21.15.132:0/4243302822 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7ff0dc02fd50 0x7ff0dc032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff0f00096a0 tx=0x7ff0f0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:22.911 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.908+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/4243302822 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f4101db0 con 0x7ff0f40703b0 2023-12-06T03:52:22.914 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:22.913+0000 7ff0e27fc700 1 -- 172.21.15.132:0/4243302822 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff0e4016070 con 0x7ff0f40703b0 2023-12-06T03:52:23.090 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.088+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/4243302822 --> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0f4101f70 con 0x7ff0dc02fd50 2023-12-06T03:52:23.095 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.093+0000 7ff0e27fc700 1 -- 172.21.15.132:0/4243302822 <== mgr.14114 v2:172.21.15.132:6800/3766660979 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff0f4101f70 con 0x7ff0dc02fd50 2023-12-06T03:52:23.098 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.095+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/4243302822 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7ff0dc02fd50 msgr2=0x7ff0dc032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:23.098 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.095+0000 7ff0fc1e3700 1 --2- 172.21.15.132:0/4243302822 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7ff0dc02fd50 0x7ff0dc032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff0f00096a0 tx=0x7ff0f0006b40).stop 2023-12-06T03:52:23.098 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.095+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/4243302822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0f40703b0 msgr2=0x7ff0f410ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:23.098 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.095+0000 7ff0fc1e3700 1 --2- 172.21.15.132:0/4243302822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0f40703b0 0x7ff0f410ccf0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7ff0e400e040 tx=0x7ff0e4006ee0).stop 2023-12-06T03:52:23.098 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.095+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/4243302822 shutdown_connections 2023-12-06T03:52:23.098 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.095+0000 7ff0fc1e3700 1 --2- 172.21.15.132:0/4243302822 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7ff0dc02fd50 0x7ff0dc032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:23.098 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.095+0000 7ff0fc1e3700 1 --2- 172.21.15.132:0/4243302822 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0f40703b0 0x7ff0f410ccf0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:23.098 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.095+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/4243302822 >> 172.21.15.132:0/4243302822 conn(0x7ff0f40f63c0 msgr2=0x7ff0f40f7f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:23.099 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.095+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/4243302822 shutdown_connections 2023-12-06T03:52:23.099 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.095+0000 7ff0fc1e3700 1 -- 172.21.15.132:0/4243302822 wait complete. 2023-12-06T03:52:23.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:22 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:21.753147+0000 mgr.smithi132.gokxja (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-06T03:52:23.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:22 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:21.753812+0000 mon.smithi132 (mon.0) 47 : cluster [DBG] mgrmap e6: smithi132.gokxja(active, since 1.05813s) 2023-12-06T03:52:23.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:22 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:21.759833+0000 mgr.smithi132.gokxja (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-06T03:52:23.439 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-12-06T03:52:23.792 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.790+0000 7fe4c34c7700 1 Processor -- start 2023-12-06T03:52:23.792 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.790+0000 7fe4c34c7700 1 -- start start 2023-12-06T03:52:23.792 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.790+0000 7fe4c34c7700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4bc0703f0 0x7fe4bc0fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:23.792 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.790+0000 7fe4c34c7700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc0fec60 con 0x7fe4bc0703f0 2023-12-06T03:52:23.792 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.790+0000 7fe4c24c5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4bc0703f0 0x7fe4bc0fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:23.792 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.790+0000 7fe4c24c5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4bc0703f0 0x7fe4bc0fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:50134/0 (socket says 172.21.15.132:50134) 2023-12-06T03:52:23.792 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.790+0000 7fe4c24c5700 1 -- 172.21.15.132:0/3608666716 learned_addr learned my addr 172.21.15.132:0/3608666716 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:23.793 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.790+0000 7fe4c24c5700 1 -- 172.21.15.132:0/3608666716 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4bc0feda0 con 0x7fe4bc0703f0 2023-12-06T03:52:23.793 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.790+0000 7fe4c24c5700 1 --2- 172.21.15.132:0/3608666716 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4bc0703f0 0x7fe4bc0fe770 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe4b800b5f0 tx=0x7fe4b8006ee0).ready entity=mon.0 client_cookie=bd41f3cb74d46aa5 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:23.793 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.790+0000 7fe4c14c3700 1 -- 172.21.15.132:0/3608666716 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4b800d600 con 0x7fe4bc0703f0 2023-12-06T03:52:23.793 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.790+0000 7fe4c14c3700 1 -- 172.21.15.132:0/3608666716 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe4b800dbd0 con 0x7fe4bc0703f0 2023-12-06T03:52:23.793 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.790+0000 7fe4c14c3700 1 -- 172.21.15.132:0/3608666716 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4b8011c00 con 0x7fe4bc0703f0 2023-12-06T03:52:23.796 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.790+0000 7fe4c34c7700 1 -- 172.21.15.132:0/3608666716 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4bc0703f0 msgr2=0x7fe4bc0fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:23.796 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.790+0000 7fe4c34c7700 1 --2- 172.21.15.132:0/3608666716 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4bc0703f0 0x7fe4bc0fe770 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fe4b800b5f0 tx=0x7fe4b8006ee0).stop 2023-12-06T03:52:23.796 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.791+0000 7fe4c34c7700 1 -- 172.21.15.132:0/3608666716 shutdown_connections 2023-12-06T03:52:23.796 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.791+0000 7fe4c34c7700 1 --2- 172.21.15.132:0/3608666716 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4bc0703f0 0x7fe4bc0fe770 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:23.796 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.791+0000 7fe4c34c7700 1 -- 172.21.15.132:0/3608666716 >> 172.21.15.132:0/3608666716 conn(0x7fe4bc0f6210 msgr2=0x7fe4bc0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:23.796 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.791+0000 7fe4c34c7700 1 -- 172.21.15.132:0/3608666716 shutdown_connections 2023-12-06T03:52:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.791+0000 7fe4c34c7700 1 -- 172.21.15.132:0/3608666716 wait complete. 2023-12-06T03:52:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.791+0000 7fe4c34c7700 1 Processor -- start 2023-12-06T03:52:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.791+0000 7fe4c34c7700 1 -- start start 2023-12-06T03:52:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.791+0000 7fe4c34c7700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4bc0703f0 0x7fe4bc10a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.791+0000 7fe4c34c7700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4bc10ad00 con 0x7fe4bc0703f0 2023-12-06T03:52:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.791+0000 7fe4c24c5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4bc0703f0 0x7fe4bc10a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.792+0000 7fe4c24c5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4bc0703f0 0x7fe4bc10a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:50138/0 (socket says 172.21.15.132:50138) 2023-12-06T03:52:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.792+0000 7fe4c24c5700 1 -- 172.21.15.132:0/2370908996 learned_addr learned my addr 172.21.15.132:0/2370908996 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.792+0000 7fe4c24c5700 1 -- 172.21.15.132:0/2370908996 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4b800b200 con 0x7fe4bc0703f0 2023-12-06T03:52:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.792+0000 7fe4c24c5700 1 --2- 172.21.15.132:0/2370908996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4bc0703f0 0x7fe4bc10a810 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe4b800b190 tx=0x7fe4b8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.792+0000 7fe4aaffd700 1 -- 172.21.15.132:0/2370908996 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4b801b410 con 0x7fe4bc0703f0 2023-12-06T03:52:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.792+0000 7fe4aaffd700 1 -- 172.21.15.132:0/2370908996 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe4b801b9e0 con 0x7fe4bc0703f0 2023-12-06T03:52:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.792+0000 7fe4aaffd700 1 -- 172.21.15.132:0/2370908996 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4b801a880 con 0x7fe4bc0703f0 2023-12-06T03:52:23.797 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.792+0000 7fe4c34c7700 1 -- 172.21.15.132:0/2370908996 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4bc10af00 con 0x7fe4bc0703f0 2023-12-06T03:52:23.798 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.792+0000 7fe4c34c7700 1 -- 172.21.15.132:0/2370908996 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4bc10b310 con 0x7fe4bc0703f0 2023-12-06T03:52:23.798 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.793+0000 7fe4aaffd700 1 -- 172.21.15.132:0/2370908996 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe4b800f070 con 0x7fe4bc0703f0 2023-12-06T03:52:23.798 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.793+0000 7fe4c34c7700 1 -- 172.21.15.132:0/2370908996 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4bc045ad0 con 0x7fe4bc0703f0 2023-12-06T03:52:23.798 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.793+0000 7fe4aaffd700 1 --2- 172.21.15.132:0/2370908996 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fe4a402fdb0 0x7fe4a4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:23.798 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.793+0000 7fe4aaffd700 1 -- 172.21.15.132:0/2370908996 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe4b80411e0 con 0x7fe4bc0703f0 2023-12-06T03:52:23.798 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.793+0000 7fe4c1cc4700 1 --2- 172.21.15.132:0/2370908996 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fe4a402fdb0 0x7fe4a4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:23.798 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.793+0000 7fe4c1cc4700 1 --2- 172.21.15.132:0/2370908996 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fe4a402fdb0 0x7fe4a4032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe4ac0096a0 tx=0x7fe4ac006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:23.799 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.798+0000 7fe4aaffd700 1 -- 172.21.15.132:0/2370908996 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4b8019950 con 0x7fe4bc0703f0 2023-12-06T03:52:23.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:22.754389+0000 mon.smithi132 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi132.gokxja(active, since 2s) 2023-12-06T03:52:23.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:23.094786+0000 mon.smithi132 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:23.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:23.097910+0000 mon.smithi132 (mon.0) 50 : audit [DBG] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-06T03:52:23.952 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.950+0000 7fe4c34c7700 1 -- 172.21.15.132:0/2370908996 --> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fe4bc10bbf0 con 0x7fe4a402fdb0 2023-12-06T03:52:23.952 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.951+0000 7fe4aaffd700 1 -- 172.21.15.132:0/2370908996 <== mgr.14114 v2:172.21.15.132:6800/3766660979 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fe4bc10bbf0 con 0x7fe4a402fdb0 2023-12-06T03:52:23.952 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: value unchanged 2023-12-06T03:52:23.954 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.952+0000 7fe4c34c7700 1 -- 172.21.15.132:0/2370908996 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fe4a402fdb0 msgr2=0x7fe4a4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:23.954 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.952+0000 7fe4c34c7700 1 --2- 172.21.15.132:0/2370908996 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fe4a402fdb0 0x7fe4a4032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe4ac0096a0 tx=0x7fe4ac006b40).stop 2023-12-06T03:52:23.954 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.952+0000 7fe4c34c7700 1 -- 172.21.15.132:0/2370908996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4bc0703f0 msgr2=0x7fe4bc10a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:23.954 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.952+0000 7fe4c34c7700 1 --2- 172.21.15.132:0/2370908996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4bc0703f0 0x7fe4bc10a810 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fe4b800b190 tx=0x7fe4b8006ee0).stop 2023-12-06T03:52:23.954 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.952+0000 7fe4c34c7700 1 -- 172.21.15.132:0/2370908996 shutdown_connections 2023-12-06T03:52:23.954 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.952+0000 7fe4c34c7700 1 --2- 172.21.15.132:0/2370908996 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fe4a402fdb0 0x7fe4a4032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:23.954 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.952+0000 7fe4c34c7700 1 --2- 172.21.15.132:0/2370908996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe4bc0703f0 0x7fe4bc10a810 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:23.954 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.952+0000 7fe4c34c7700 1 -- 172.21.15.132:0/2370908996 >> 172.21.15.132:0/2370908996 conn(0x7fe4bc0f6210 msgr2=0x7fe4bc0f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:23.954 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.952+0000 7fe4c34c7700 1 -- 172.21.15.132:0/2370908996 shutdown_connections 2023-12-06T03:52:23.955 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:23.952+0000 7fe4c34c7700 1 -- 172.21.15.132:0/2370908996 wait complete. 2023-12-06T03:52:24.307 INFO:teuthology.orchestra.run.smithi132.stderr:Generating ssh key... 2023-12-06T03:52:24.307 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-12-06T03:52:24.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.633+0000 7ff8bd8f1700 1 Processor -- start 2023-12-06T03:52:24.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.633+0000 7ff8bd8f1700 1 -- start start 2023-12-06T03:52:24.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.633+0000 7ff8bd8f1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8b8072560 0x7ff8b8070b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:24.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.633+0000 7ff8bd8f1700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b8072980 con 0x7ff8b8072560 2023-12-06T03:52:24.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.633+0000 7ff8b6ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8b8072560 0x7ff8b8070b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:24.636 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.633+0000 7ff8b6ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8b8072560 0x7ff8b8070b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:50152/0 (socket says 172.21.15.132:50152) 2023-12-06T03:52:24.636 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.633+0000 7ff8b6ffd700 1 -- 172.21.15.132:0/878222666 learned_addr learned my addr 172.21.15.132:0/878222666 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:24.636 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.633+0000 7ff8b6ffd700 1 -- 172.21.15.132:0/878222666 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8b8070ff0 con 0x7ff8b8072560 2023-12-06T03:52:24.636 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.633+0000 7ff8b6ffd700 1 --2- 172.21.15.132:0/878222666 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8b8072560 0x7ff8b8070b00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff8a800b5f0 tx=0x7ff8a8006ee0).ready entity=mon.0 client_cookie=61b5f7eadc7855c1 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:24.636 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.634+0000 7ff8b5ffb700 1 -- 172.21.15.132:0/878222666 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8a800d600 con 0x7ff8b8072560 2023-12-06T03:52:24.636 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.634+0000 7ff8b5ffb700 1 -- 172.21.15.132:0/878222666 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff8a800dbd0 con 0x7ff8b8072560 2023-12-06T03:52:24.636 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.634+0000 7ff8b5ffb700 1 -- 172.21.15.132:0/878222666 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8a8011c00 con 0x7ff8b8072560 2023-12-06T03:52:24.636 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.634+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/878222666 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8b8072560 msgr2=0x7ff8b8070b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:24.639 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.634+0000 7ff8bd8f1700 1 --2- 172.21.15.132:0/878222666 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8b8072560 0x7ff8b8070b00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff8a800b5f0 tx=0x7ff8a8006ee0).stop 2023-12-06T03:52:24.639 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.634+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/878222666 shutdown_connections 2023-12-06T03:52:24.639 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.634+0000 7ff8bd8f1700 1 --2- 172.21.15.132:0/878222666 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8b8072560 0x7ff8b8070b00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:24.639 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.634+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/878222666 >> 172.21.15.132:0/878222666 conn(0x7ff8b80f7350 msgr2=0x7ff8b80f9770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:24.639 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.634+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/878222666 shutdown_connections 2023-12-06T03:52:24.639 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.635+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/878222666 wait complete. 2023-12-06T03:52:24.639 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.635+0000 7ff8bd8f1700 1 Processor -- start 2023-12-06T03:52:24.639 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.635+0000 7ff8bd8f1700 1 -- start start 2023-12-06T03:52:24.639 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.635+0000 7ff8bd8f1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8b8072560 0x7ff8b810cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:24.639 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.635+0000 7ff8bd8f1700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8b810d000 con 0x7ff8b8072560 2023-12-06T03:52:24.639 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.635+0000 7ff8b6ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8b8072560 0x7ff8b810cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:24.640 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.636+0000 7ff8b6ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8b8072560 0x7ff8b810cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:50158/0 (socket says 172.21.15.132:50158) 2023-12-06T03:52:24.640 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.636+0000 7ff8b6ffd700 1 -- 172.21.15.132:0/3605714954 learned_addr learned my addr 172.21.15.132:0/3605714954 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:24.640 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.636+0000 7ff8b6ffd700 1 -- 172.21.15.132:0/3605714954 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a800b200 con 0x7ff8b8072560 2023-12-06T03:52:24.640 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.636+0000 7ff8b6ffd700 1 --2- 172.21.15.132:0/3605714954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8b8072560 0x7ff8b810cb10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff8a800b190 tx=0x7ff8a8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:24.640 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.636+0000 7ff89f7fe700 1 -- 172.21.15.132:0/3605714954 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8a8011a20 con 0x7ff8b8072560 2023-12-06T03:52:24.640 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.636+0000 7ff89f7fe700 1 -- 172.21.15.132:0/3605714954 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7ff8a801b880 con 0x7ff8b8072560 2023-12-06T03:52:24.640 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.636+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/3605714954 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8b810d200 con 0x7ff8b8072560 2023-12-06T03:52:24.640 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.636+0000 7ff89f7fe700 1 -- 172.21.15.132:0/3605714954 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff8a801a710 con 0x7ff8b8072560 2023-12-06T03:52:24.640 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.636+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/3605714954 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8b810d610 con 0x7ff8b8072560 2023-12-06T03:52:24.640 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.637+0000 7ff89f7fe700 1 -- 172.21.15.132:0/3605714954 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7ff8a800f070 con 0x7ff8b8072560 2023-12-06T03:52:24.640 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.637+0000 7ff89f7fe700 1 --2- 172.21.15.132:0/3605714954 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7ff8a402fd50 0x7ff8a4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:24.640 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.637+0000 7ff89f7fe700 1 -- 172.21.15.132:0/3605714954 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7ff8a8041120 con 0x7ff8b8072560 2023-12-06T03:52:24.640 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.637+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/3605714954 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8b8045ad0 con 0x7ff8b8072560 2023-12-06T03:52:24.641 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.637+0000 7ff8b67fc700 1 --2- 172.21.15.132:0/3605714954 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7ff8a402fd50 0x7ff8a4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:24.641 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.638+0000 7ff8b67fc700 1 --2- 172.21.15.132:0/3605714954 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7ff8a402fd50 0x7ff8a4032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff8ac0096a0 tx=0x7ff8ac006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:24.642 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.641+0000 7ff89f7fe700 1 -- 172.21.15.132:0/3605714954 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8a8025460 con 0x7ff8b8072560 2023-12-06T03:52:24.795 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:24.794+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/3605714954 --> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff8b810def0 con 0x7ff8a402fd50 2023-12-06T03:52:25.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:24 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:23.090886+0000 mgr.smithi132.gokxja (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-12-06T03:52:25.218 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.217+0000 7ff89f7fe700 1 -- 172.21.15.132:0/3605714954 <== mgr.14114 v2:172.21.15.132:6800/3766660979 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff8b810def0 con 0x7ff8a402fd50 2023-12-06T03:52:25.220 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.219+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/3605714954 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7ff8a402fd50 msgr2=0x7ff8a4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:25.220 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.219+0000 7ff8bd8f1700 1 --2- 172.21.15.132:0/3605714954 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7ff8a402fd50 0x7ff8a4032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7ff8ac0096a0 tx=0x7ff8ac006b40).stop 2023-12-06T03:52:25.221 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.219+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/3605714954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8b8072560 msgr2=0x7ff8b810cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:25.221 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.219+0000 7ff8bd8f1700 1 --2- 172.21.15.132:0/3605714954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8b8072560 0x7ff8b810cb10 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7ff8a800b190 tx=0x7ff8a8006ee0).stop 2023-12-06T03:52:25.221 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.219+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/3605714954 shutdown_connections 2023-12-06T03:52:25.221 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.219+0000 7ff8bd8f1700 1 --2- 172.21.15.132:0/3605714954 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7ff8a402fd50 0x7ff8a4032210 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:25.221 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.219+0000 7ff8bd8f1700 1 --2- 172.21.15.132:0/3605714954 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff8b8072560 0x7ff8b810cb10 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:25.221 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.219+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/3605714954 >> 172.21.15.132:0/3605714954 conn(0x7ff8b80f7350 msgr2=0x7ff8b80f8eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:25.221 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.219+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/3605714954 shutdown_connections 2023-12-06T03:52:25.221 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.219+0000 7ff8bd8f1700 1 -- 172.21.15.132:0/3605714954 wait complete. 2023-12-06T03:52:25.564 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-12-06T03:52:25.903 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.898+0000 7f01dca68700 1 Processor -- start 2023-12-06T03:52:25.903 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.899+0000 7f01dca68700 1 -- start start 2023-12-06T03:52:25.903 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.899+0000 7f01dca68700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01d80fab80 0x7f01d80fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:25.903 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.899+0000 7f01dca68700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d80fb490 con 0x7f01d80fab80 2023-12-06T03:52:25.903 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.899+0000 7f01d659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01d80fab80 0x7f01d80fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:25.903 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.899+0000 7f01d659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01d80fab80 0x7f01d80fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:50168/0 (socket says 172.21.15.132:50168) 2023-12-06T03:52:25.903 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.899+0000 7f01d659c700 1 -- 172.21.15.132:0/1887397801 learned_addr learned my addr 172.21.15.132:0/1887397801 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:25.903 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.899+0000 7f01d659c700 1 -- 172.21.15.132:0/1887397801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d80fb5d0 con 0x7f01d80fab80 2023-12-06T03:52:25.904 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.899+0000 7f01d659c700 1 --2- 172.21.15.132:0/1887397801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01d80fab80 0x7f01d80fafa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f01c000b5f0 tx=0x7f01c0006ee0).ready entity=mon.0 client_cookie=cdbbb092839f655e server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:25.904 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.900+0000 7f01d559a700 1 -- 172.21.15.132:0/1887397801 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c000d600 con 0x7f01d80fab80 2023-12-06T03:52:25.904 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.900+0000 7f01d559a700 1 -- 172.21.15.132:0/1887397801 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f01c000dbd0 con 0x7f01d80fab80 2023-12-06T03:52:25.904 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.900+0000 7f01d559a700 1 -- 172.21.15.132:0/1887397801 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c0011c00 con 0x7f01d80fab80 2023-12-06T03:52:25.904 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.900+0000 7f01dca68700 1 -- 172.21.15.132:0/1887397801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01d80fab80 msgr2=0x7f01d80fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:25.904 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.900+0000 7f01dca68700 1 --2- 172.21.15.132:0/1887397801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01d80fab80 0x7f01d80fafa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f01c000b5f0 tx=0x7f01c0006ee0).stop 2023-12-06T03:52:25.904 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.900+0000 7f01dca68700 1 -- 172.21.15.132:0/1887397801 shutdown_connections 2023-12-06T03:52:25.904 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.900+0000 7f01dca68700 1 --2- 172.21.15.132:0/1887397801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01d80fab80 0x7f01d80fafa0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:25.904 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.900+0000 7f01dca68700 1 -- 172.21.15.132:0/1887397801 >> 172.21.15.132:0/1887397801 conn(0x7f01d80f63c0 msgr2=0x7f01d80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:25.904 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.900+0000 7f01dca68700 1 -- 172.21.15.132:0/1887397801 shutdown_connections 2023-12-06T03:52:25.904 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.900+0000 7f01dca68700 1 -- 172.21.15.132:0/1887397801 wait complete. 2023-12-06T03:52:25.904 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.901+0000 7f01dca68700 1 Processor -- start 2023-12-06T03:52:25.904 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.901+0000 7f01dca68700 1 -- start start 2023-12-06T03:52:25.904 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.901+0000 7f01dca68700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01d80fab80 0x7f01d810ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:25.905 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.901+0000 7f01dca68700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01d810d1b0 con 0x7f01d80fab80 2023-12-06T03:52:25.905 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.901+0000 7f01d659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01d80fab80 0x7f01d810ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:25.905 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.901+0000 7f01d659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01d80fab80 0x7f01d810ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:50182/0 (socket says 172.21.15.132:50182) 2023-12-06T03:52:25.905 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.901+0000 7f01d659c700 1 -- 172.21.15.132:0/2626681026 learned_addr learned my addr 172.21.15.132:0/2626681026 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:25.905 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.901+0000 7f01d659c700 1 -- 172.21.15.132:0/2626681026 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01c000b200 con 0x7f01d80fab80 2023-12-06T03:52:25.905 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.901+0000 7f01d659c700 1 --2- 172.21.15.132:0/2626681026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01d80fab80 0x7f01d810ccc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f01c000e040 tx=0x7f01c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:25.905 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.902+0000 7f01ceffd700 1 -- 172.21.15.132:0/2626681026 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c001b410 con 0x7f01d80fab80 2023-12-06T03:52:25.905 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.902+0000 7f01ceffd700 1 -- 172.21.15.132:0/2626681026 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f01c001b9e0 con 0x7f01d80fab80 2023-12-06T03:52:25.905 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.902+0000 7f01dca68700 1 -- 172.21.15.132:0/2626681026 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01d810d3b0 con 0x7f01d80fab80 2023-12-06T03:52:25.905 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.902+0000 7f01ceffd700 1 -- 172.21.15.132:0/2626681026 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01c001a820 con 0x7f01d80fab80 2023-12-06T03:52:25.905 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.902+0000 7f01dca68700 1 -- 172.21.15.132:0/2626681026 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01d810d7c0 con 0x7f01d80fab80 2023-12-06T03:52:25.906 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.902+0000 7f01ceffd700 1 -- 172.21.15.132:0/2626681026 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f01c000f070 con 0x7f01d80fab80 2023-12-06T03:52:25.907 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.903+0000 7f01dca68700 1 -- 172.21.15.132:0/2626681026 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01d8045ad0 con 0x7f01d80fab80 2023-12-06T03:52:25.907 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.903+0000 7f01ceffd700 1 --2- 172.21.15.132:0/2626681026 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f01b802fd40 0x7f01b8032200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:25.907 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.903+0000 7f01ceffd700 1 -- 172.21.15.132:0/2626681026 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f01c0041450 con 0x7f01d80fab80 2023-12-06T03:52:25.907 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.903+0000 7f01d5d9b700 1 --2- 172.21.15.132:0/2626681026 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f01b802fd40 0x7f01b8032200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:25.907 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.903+0000 7f01d5d9b700 1 --2- 172.21.15.132:0/2626681026 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f01b802fd40 0x7f01b8032200 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f01c80096a0 tx=0x7f01c8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:25.907 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:25.907+0000 7f01ceffd700 1 -- 172.21.15.132:0/2626681026 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f01c0022b70 con 0x7f01d80fab80 2023-12-06T03:52:26.057 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.056+0000 7f01dca68700 1 -- 172.21.15.132:0/2626681026 --> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f01d810e020 con 0x7f01b802fd40 2023-12-06T03:52:26.057 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.057+0000 7f01ceffd700 1 -- 172.21.15.132:0/2626681026 <== mgr.14114 v2:172.21.15.132:6800/3766660979 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f01d810e020 con 0x7f01b802fd40 2023-12-06T03:52:26.058 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmtWJq0ND9ami/WEfjlFjk0GN9nrC6a7gbEGPXxm284p1GEkAT8/urB9hxmKS1PO0m59Fl/ePaPd0G3skDGMiJBg7axICw6Du0qD4ffgVWm9IxPc5pNOU88wj30TL5ksttzh+Ju8/hUva2mRgwFAZOuS0yfH9wwy5TtHHQ5uFBt8xnghWV7oRMGA8U236rtvR9RLPxfky6bhiPqyow3JBX2CIjzT8uD68n3VqE8w4GdZCeN59BV1i+dl5ihdioUP6XzlTg8DyncNgZLzle0s4XuCPt2xYxucfEB9/B2QoMJHQOiB4kRcniP8uFIqLiobF7ahL6ge96YiSQNKGWrZyHk1Y9/L3XlBv+MfRURM+9MQ8vFQZes3UwfMEoHwnHs0GYpuLgWM1Gevuh3XTCLrhIq2vHzvYVV0zXjwooHA2ryvtR2IEEjNDnmEnPP95l0gb1cfqmbnb/9JGh6GlBX6QmJDnxcHzQF5/v6/N4P3U+VMTD8yeNjh/toMVOFBe5frE= ceph-c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:52:26.060 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.058+0000 7f01dca68700 1 -- 172.21.15.132:0/2626681026 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f01b802fd40 msgr2=0x7f01b8032200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:26.060 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.058+0000 7f01dca68700 1 --2- 172.21.15.132:0/2626681026 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f01b802fd40 0x7f01b8032200 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f01c80096a0 tx=0x7f01c8006b40).stop 2023-12-06T03:52:26.060 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.058+0000 7f01dca68700 1 -- 172.21.15.132:0/2626681026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01d80fab80 msgr2=0x7f01d810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:26.060 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.058+0000 7f01dca68700 1 --2- 172.21.15.132:0/2626681026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01d80fab80 0x7f01d810ccc0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f01c000e040 tx=0x7f01c0006ee0).stop 2023-12-06T03:52:26.060 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.059+0000 7f01dca68700 1 -- 172.21.15.132:0/2626681026 shutdown_connections 2023-12-06T03:52:26.061 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.059+0000 7f01dca68700 1 --2- 172.21.15.132:0/2626681026 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f01b802fd40 0x7f01b8032200 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:26.061 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.059+0000 7f01dca68700 1 --2- 172.21.15.132:0/2626681026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f01d80fab80 0x7f01d810ccc0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:26.061 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.059+0000 7f01dca68700 1 -- 172.21.15.132:0/2626681026 >> 172.21.15.132:0/2626681026 conn(0x7f01d80f63c0 msgr2=0x7f01d80ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:26.061 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.059+0000 7f01dca68700 1 -- 172.21.15.132:0/2626681026 shutdown_connections 2023-12-06T03:52:26.061 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.059+0000 7f01dca68700 1 -- 172.21.15.132:0/2626681026 wait complete. 2023-12-06T03:52:26.082 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:23.951400+0000 mgr.smithi132.gokxja (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-12-06T03:52:26.082 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:25.215041+0000 mon.smithi132 (mon.0) 51 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:26.082 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:25.216666+0000 mon.smithi132 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:26.415 INFO:teuthology.orchestra.run.smithi132.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-12-06T03:52:26.415 INFO:teuthology.orchestra.run.smithi132.stderr:Adding key to root@localhost authorized_keys... 2023-12-06T03:52:26.416 INFO:teuthology.orchestra.run.smithi132.stderr:Adding host smithi132... 2023-12-06T03:52:26.416 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi132 172.21.15.132 2023-12-06T03:52:26.782 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.778+0000 7f550d5ce700 1 Processor -- start 2023-12-06T03:52:26.782 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.779+0000 7f550d5ce700 1 -- start start 2023-12-06T03:52:26.782 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.779+0000 7f550d5ce700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f55080703f0 0x7f55080fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:26.782 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.779+0000 7f550d5ce700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55080fec60 con 0x7f55080703f0 2023-12-06T03:52:26.782 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.779+0000 7f5507fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f55080703f0 0x7f55080fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:26.782 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.779+0000 7f5507fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f55080703f0 0x7f55080fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:50190/0 (socket says 172.21.15.132:50190) 2023-12-06T03:52:26.782 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.779+0000 7f5507fff700 1 -- 172.21.15.132:0/2656205269 learned_addr learned my addr 172.21.15.132:0/2656205269 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:26.782 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.779+0000 7f5507fff700 1 -- 172.21.15.132:0/2656205269 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f55080feda0 con 0x7f55080703f0 2023-12-06T03:52:26.782 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.779+0000 7f5507fff700 1 --2- 172.21.15.132:0/2656205269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f55080703f0 0x7f55080fe770 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f54f800b5f0 tx=0x7f54f8006ee0).ready entity=mon.0 client_cookie=96994b2879c7d93a server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:26.782 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.779+0000 7f5506ffd700 1 -- 172.21.15.132:0/2656205269 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54f800d600 con 0x7f55080703f0 2023-12-06T03:52:26.783 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.779+0000 7f5506ffd700 1 -- 172.21.15.132:0/2656205269 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f54f800dbd0 con 0x7f55080703f0 2023-12-06T03:52:26.783 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.779+0000 7f5506ffd700 1 -- 172.21.15.132:0/2656205269 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54f8011c00 con 0x7f55080703f0 2023-12-06T03:52:26.783 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.780+0000 7f550d5ce700 1 -- 172.21.15.132:0/2656205269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f55080703f0 msgr2=0x7f55080fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:26.783 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.780+0000 7f550d5ce700 1 --2- 172.21.15.132:0/2656205269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f55080703f0 0x7f55080fe770 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f54f800b5f0 tx=0x7f54f8006ee0).stop 2023-12-06T03:52:26.783 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.780+0000 7f550d5ce700 1 -- 172.21.15.132:0/2656205269 shutdown_connections 2023-12-06T03:52:26.783 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.780+0000 7f550d5ce700 1 --2- 172.21.15.132:0/2656205269 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f55080703f0 0x7f55080fe770 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:26.783 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.780+0000 7f550d5ce700 1 -- 172.21.15.132:0/2656205269 >> 172.21.15.132:0/2656205269 conn(0x7f55080f6210 msgr2=0x7f55080f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:26.783 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.780+0000 7f550d5ce700 1 -- 172.21.15.132:0/2656205269 shutdown_connections 2023-12-06T03:52:26.783 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.780+0000 7f550d5ce700 1 -- 172.21.15.132:0/2656205269 wait complete. 2023-12-06T03:52:26.783 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.781+0000 7f550d5ce700 1 Processor -- start 2023-12-06T03:52:26.786 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.781+0000 7f550d5ce700 1 -- start start 2023-12-06T03:52:26.786 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.782+0000 7f550d5ce700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f55080703f0 0x7f550810a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:26.786 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.782+0000 7f550d5ce700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f550810ad00 con 0x7f55080703f0 2023-12-06T03:52:26.786 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.782+0000 7f5507fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f55080703f0 0x7f550810a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:26.786 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.782+0000 7f5507fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f55080703f0 0x7f550810a810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:50204/0 (socket says 172.21.15.132:50204) 2023-12-06T03:52:26.786 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.782+0000 7f5507fff700 1 -- 172.21.15.132:0/3934348836 learned_addr learned my addr 172.21.15.132:0/3934348836 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:26.786 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.782+0000 7f5507fff700 1 -- 172.21.15.132:0/3934348836 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54f800b200 con 0x7f55080703f0 2023-12-06T03:52:26.786 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.782+0000 7f5507fff700 1 --2- 172.21.15.132:0/3934348836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f55080703f0 0x7f550810a810 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f54f800b190 tx=0x7f54f8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:26.787 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.782+0000 7f5504ff9700 1 -- 172.21.15.132:0/3934348836 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54f801b410 con 0x7f55080703f0 2023-12-06T03:52:26.787 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.782+0000 7f550d5ce700 1 -- 172.21.15.132:0/3934348836 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f550810af00 con 0x7f55080703f0 2023-12-06T03:52:26.787 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.782+0000 7f5504ff9700 1 -- 172.21.15.132:0/3934348836 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f54f801b9e0 con 0x7f55080703f0 2023-12-06T03:52:26.787 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.782+0000 7f5504ff9700 1 -- 172.21.15.132:0/3934348836 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54f801a880 con 0x7f55080703f0 2023-12-06T03:52:26.787 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.782+0000 7f550d5ce700 1 -- 172.21.15.132:0/3934348836 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f550810b310 con 0x7f55080703f0 2023-12-06T03:52:26.787 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.783+0000 7f5504ff9700 1 -- 172.21.15.132:0/3934348836 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f54f800f070 con 0x7f55080703f0 2023-12-06T03:52:26.787 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.783+0000 7f5504ff9700 1 --2- 172.21.15.132:0/3934348836 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f54f402fd50 0x7f54f4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:26.787 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.783+0000 7f5504ff9700 1 -- 172.21.15.132:0/3934348836 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f54f8041380 con 0x7f55080703f0 2023-12-06T03:52:26.787 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.783+0000 7f55077fe700 1 --2- 172.21.15.132:0/3934348836 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f54f402fd50 0x7f54f4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:26.787 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.783+0000 7f550d5ce700 1 -- 172.21.15.132:0/3934348836 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5508059100 con 0x7f55080703f0 2023-12-06T03:52:26.787 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.784+0000 7f55077fe700 1 --2- 172.21.15.132:0/3934348836 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f54f402fd50 0x7f54f4032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f54fc0096a0 tx=0x7f54fc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:26.790 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.788+0000 7f5504ff9700 1 -- 172.21.15.132:0/3934348836 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f54f8010360 con 0x7f55080703f0 2023-12-06T03:52:26.961 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:26.960+0000 7f550d5ce700 1 -- 172.21.15.132:0/3934348836 --> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi132", "addr": "172.21.15.132", "target": ["mon-mgr", ""]}) v1 -- 0x7f5508104740 con 0x7f54f402fd50 2023-12-06T03:52:27.061 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:26 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:52:24.795221+0000 mgr.smithi132.gokxja (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2023-12-06T03:52:27.602 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: Added host 'smithi132' 2023-12-06T03:52:27.602 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:27.600+0000 7f5504ff9700 1 -- 172.21.15.132:0/3934348836 <== mgr.14114 v2:172.21.15.132:6800/3766660979 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f5508104740 con 0x7f54f402fd50 2023-12-06T03:52:27.606 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:27.603+0000 7f550d5ce700 1 -- 172.21.15.132:0/3934348836 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f54f402fd50 msgr2=0x7f54f4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:27.606 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:27.603+0000 7f550d5ce700 1 --2- 172.21.15.132:0/3934348836 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f54f402fd50 0x7f54f4032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f54fc0096a0 tx=0x7f54fc006b40).stop 2023-12-06T03:52:27.606 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:27.603+0000 7f550d5ce700 1 -- 172.21.15.132:0/3934348836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f55080703f0 msgr2=0x7f550810a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:27.606 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:27.603+0000 7f550d5ce700 1 --2- 172.21.15.132:0/3934348836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f55080703f0 0x7f550810a810 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f54f800b190 tx=0x7f54f8006ee0).stop 2023-12-06T03:52:27.606 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:27.603+0000 7f550d5ce700 1 -- 172.21.15.132:0/3934348836 shutdown_connections 2023-12-06T03:52:27.606 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:27.603+0000 7f550d5ce700 1 --2- 172.21.15.132:0/3934348836 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f54f402fd50 0x7f54f4032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:27.607 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:27.603+0000 7f550d5ce700 1 --2- 172.21.15.132:0/3934348836 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f55080703f0 0x7f550810a810 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:27.607 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:27.603+0000 7f550d5ce700 1 -- 172.21.15.132:0/3934348836 >> 172.21.15.132:0/3934348836 conn(0x7f55080f6210 msgr2=0x7f55080f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:27.607 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:27.604+0000 7f550d5ce700 1 -- 172.21.15.132:0/3934348836 shutdown_connections 2023-12-06T03:52:27.607 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:27.604+0000 7f550d5ce700 1 -- 172.21.15.132:0/3934348836 wait complete. 2023-12-06T03:52:28.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:26.057341+0000 mgr.smithi132.gokxja (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-06T03:52:28.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:27.599845+0000 mon.smithi132 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:28.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:27.600395+0000 mon.smithi132 (mon.0) 54 : audit [DBG] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-06T03:52:29.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:28 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:26.961493+0000 mgr.smithi132.gokxja (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi132", "addr": "172.21.15.132", "target": ["mon-mgr", ""]}]: dispatch 2023-12-06T03:52:29.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:28 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:52:27.600150+0000 mgr.smithi132.gokxja (mgr.14114) 9 : cephadm [INF] Added host smithi132 2023-12-06T03:52:29.148 INFO:teuthology.orchestra.run.smithi132.stderr:Deploying mon service with default placement... 2023-12-06T03:52:29.148 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-12-06T03:52:30.693 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.519+0000 7f5eabbee700 1 Processor -- start 2023-12-06T03:52:30.694 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.519+0000 7f5eabbee700 1 -- start start 2023-12-06T03:52:30.694 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.519+0000 7f5eabbee700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea40fab80 0x7f5ea40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:30.694 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.519+0000 7f5eabbee700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea40fb490 con 0x7f5ea40fab80 2023-12-06T03:52:30.694 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.520+0000 7f5ea998a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea40fab80 0x7f5ea40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:30.694 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.520+0000 7f5ea998a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea40fab80 0x7f5ea40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:58464/0 (socket says 172.21.15.132:58464) 2023-12-06T03:52:30.694 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.520+0000 7f5ea998a700 1 -- 172.21.15.132:0/3440861152 learned_addr learned my addr 172.21.15.132:0/3440861152 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:30.694 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.520+0000 7f5ea998a700 1 -- 172.21.15.132:0/3440861152 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea40fb5d0 con 0x7f5ea40fab80 2023-12-06T03:52:30.694 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.520+0000 7f5ea998a700 1 --2- 172.21.15.132:0/3440861152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea40fab80 0x7f5ea40fafa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f5ea000b5f0 tx=0x7f5ea0006ee0).ready entity=mon.0 client_cookie=b855c87c2e6673c3 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:30.694 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.520+0000 7f5ea8988700 1 -- 172.21.15.132:0/3440861152 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ea000d600 con 0x7f5ea40fab80 2023-12-06T03:52:30.694 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.520+0000 7f5ea8988700 1 -- 172.21.15.132:0/3440861152 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5ea000dbd0 con 0x7f5ea40fab80 2023-12-06T03:52:30.694 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.520+0000 7f5ea8988700 1 -- 172.21.15.132:0/3440861152 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ea0011c00 con 0x7f5ea40fab80 2023-12-06T03:52:30.694 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.521+0000 7f5eabbee700 1 -- 172.21.15.132:0/3440861152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea40fab80 msgr2=0x7f5ea40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.521+0000 7f5eabbee700 1 --2- 172.21.15.132:0/3440861152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea40fab80 0x7f5ea40fafa0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f5ea000b5f0 tx=0x7f5ea0006ee0).stop 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.521+0000 7f5eabbee700 1 -- 172.21.15.132:0/3440861152 shutdown_connections 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.521+0000 7f5eabbee700 1 --2- 172.21.15.132:0/3440861152 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea40fab80 0x7f5ea40fafa0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.521+0000 7f5eabbee700 1 -- 172.21.15.132:0/3440861152 >> 172.21.15.132:0/3440861152 conn(0x7f5ea40f63c0 msgr2=0x7f5ea40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.521+0000 7f5eabbee700 1 -- 172.21.15.132:0/3440861152 shutdown_connections 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.521+0000 7f5eabbee700 1 -- 172.21.15.132:0/3440861152 wait complete. 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.522+0000 7f5eabbee700 1 Processor -- start 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.522+0000 7f5eabbee700 1 -- start start 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.522+0000 7f5eabbee700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea40fab80 0x7f5ea4109900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.522+0000 7f5eabbee700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea4109df0 con 0x7f5ea40fab80 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.522+0000 7f5ea998a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea40fab80 0x7f5ea4109900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.522+0000 7f5ea998a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea40fab80 0x7f5ea4109900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:58468/0 (socket says 172.21.15.132:58468) 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.522+0000 7f5ea998a700 1 -- 172.21.15.132:0/2926831046 learned_addr learned my addr 172.21.15.132:0/2926831046 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.522+0000 7f5ea998a700 1 -- 172.21.15.132:0/2926831046 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ea000b200 con 0x7f5ea40fab80 2023-12-06T03:52:30.695 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.522+0000 7f5ea998a700 1 --2- 172.21.15.132:0/2926831046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea40fab80 0x7f5ea4109900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f5ea000b1d0 tx=0x7f5ea0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:30.696 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.522+0000 7f5e927fc700 1 -- 172.21.15.132:0/2926831046 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ea001b410 con 0x7f5ea40fab80 2023-12-06T03:52:30.696 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.522+0000 7f5e927fc700 1 -- 172.21.15.132:0/2926831046 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5ea001b9e0 con 0x7f5ea40fab80 2023-12-06T03:52:30.696 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.522+0000 7f5e927fc700 1 -- 172.21.15.132:0/2926831046 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ea001a840 con 0x7f5ea40fab80 2023-12-06T03:52:30.696 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.522+0000 7f5eabbee700 1 -- 172.21.15.132:0/2926831046 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea4109ff0 con 0x7f5ea40fab80 2023-12-06T03:52:30.696 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.522+0000 7f5eabbee700 1 -- 172.21.15.132:0/2926831046 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea410a400 con 0x7f5ea40fab80 2023-12-06T03:52:30.696 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.523+0000 7f5e927fc700 1 -- 172.21.15.132:0/2926831046 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f5ea000f070 con 0x7f5ea40fab80 2023-12-06T03:52:30.696 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.523+0000 7f5e927fc700 1 --2- 172.21.15.132:0/2926831046 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f5e8c02fdb0 0x7f5e8c032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:30.696 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.523+0000 7f5eabbee700 1 -- 172.21.15.132:0/2926831046 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea4045ad0 con 0x7f5ea40fab80 2023-12-06T03:52:30.696 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.524+0000 7f5e927fc700 1 -- 172.21.15.132:0/2926831046 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f5ea0041350 con 0x7f5ea40fab80 2023-12-06T03:52:30.696 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.524+0000 7f5ea9189700 1 --2- 172.21.15.132:0/2926831046 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f5e8c02fdb0 0x7f5e8c032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:30.696 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.524+0000 7f5ea9189700 1 --2- 172.21.15.132:0/2926831046 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f5e8c02fdb0 0x7f5e8c032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5e940096a0 tx=0x7f5e94006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:30.696 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.528+0000 7f5e927fc700 1 -- 172.21.15.132:0/2926831046 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ea0019af0 con 0x7f5ea40fab80 2023-12-06T03:52:30.707 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.706+0000 7f5eabbee700 1 -- 172.21.15.132:0/2926831046 --> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f5ea410ad40 con 0x7f5e8c02fdb0 2023-12-06T03:52:30.711 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.710+0000 7f5e927fc700 1 -- 172.21.15.132:0/2926831046 <== mgr.14114 v2:172.21.15.132:6800/3766660979 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5ea410ad40 con 0x7f5e8c02fdb0 2023-12-06T03:52:30.711 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: Scheduled mon update... 2023-12-06T03:52:30.714 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.712+0000 7f5eabbee700 1 -- 172.21.15.132:0/2926831046 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f5e8c02fdb0 msgr2=0x7f5e8c032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:30.714 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.712+0000 7f5eabbee700 1 --2- 172.21.15.132:0/2926831046 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f5e8c02fdb0 0x7f5e8c032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5e940096a0 tx=0x7f5e94006b40).stop 2023-12-06T03:52:30.714 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.712+0000 7f5eabbee700 1 -- 172.21.15.132:0/2926831046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea40fab80 msgr2=0x7f5ea4109900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:30.715 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.712+0000 7f5eabbee700 1 --2- 172.21.15.132:0/2926831046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea40fab80 0x7f5ea4109900 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f5ea000b1d0 tx=0x7f5ea0006ee0).stop 2023-12-06T03:52:30.715 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.712+0000 7f5eabbee700 1 -- 172.21.15.132:0/2926831046 shutdown_connections 2023-12-06T03:52:30.715 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.712+0000 7f5eabbee700 1 --2- 172.21.15.132:0/2926831046 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f5e8c02fdb0 0x7f5e8c032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:30.715 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.712+0000 7f5eabbee700 1 --2- 172.21.15.132:0/2926831046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea40fab80 0x7f5ea4109900 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:30.715 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.712+0000 7f5eabbee700 1 -- 172.21.15.132:0/2926831046 >> 172.21.15.132:0/2926831046 conn(0x7f5ea40f63c0 msgr2=0x7f5ea40f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:30.715 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.712+0000 7f5eabbee700 1 -- 172.21.15.132:0/2926831046 shutdown_connections 2023-12-06T03:52:30.715 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:30.712+0000 7f5eabbee700 1 -- 172.21.15.132:0/2926831046 wait complete. 2023-12-06T03:52:32.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:31 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:30.707747+0000 mgr.smithi132.gokxja (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-12-06T03:52:32.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:31 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:52:30.708474+0000 mgr.smithi132.gokxja (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-12-06T03:52:32.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:31 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:30.710863+0000 mon.smithi132 (mon.0) 55 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:32.613 INFO:teuthology.orchestra.run.smithi132.stderr:Deploying mgr service with default placement... 2023-12-06T03:52:32.614 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-12-06T03:52:33.892 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.726+0000 7fbef3821700 1 Processor -- start 2023-12-06T03:52:33.892 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.726+0000 7fbef3821700 1 -- start start 2023-12-06T03:52:33.892 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.726+0000 7fbef3821700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbeec0703b0 0x7fbeec100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.726+0000 7fbef3821700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec0fc750 con 0x7fbeec0703b0 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.727+0000 7fbef15bd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbeec0703b0 0x7fbeec100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.727+0000 7fbef15bd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbeec0703b0 0x7fbeec100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:58480/0 (socket says 172.21.15.132:58480) 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.727+0000 7fbef15bd700 1 -- 172.21.15.132:0/819107883 learned_addr learned my addr 172.21.15.132:0/819107883 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.727+0000 7fbef15bd700 1 -- 172.21.15.132:0/819107883 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeec101250 con 0x7fbeec0703b0 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.727+0000 7fbef15bd700 1 --2- 172.21.15.132:0/819107883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbeec0703b0 0x7fbeec100d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fbee800b5f0 tx=0x7fbee8006ee0).ready entity=mon.0 client_cookie=dac9c459fdd040ed server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.727+0000 7fbedbfff700 1 -- 172.21.15.132:0/819107883 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbee800d600 con 0x7fbeec0703b0 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.727+0000 7fbedbfff700 1 -- 172.21.15.132:0/819107883 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbee800dbd0 con 0x7fbeec0703b0 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.727+0000 7fbedbfff700 1 -- 172.21.15.132:0/819107883 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbee8011c00 con 0x7fbeec0703b0 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.728+0000 7fbef3821700 1 -- 172.21.15.132:0/819107883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbeec0703b0 msgr2=0x7fbeec100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.728+0000 7fbef3821700 1 --2- 172.21.15.132:0/819107883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbeec0703b0 0x7fbeec100d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fbee800b5f0 tx=0x7fbee8006ee0).stop 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.728+0000 7fbef3821700 1 -- 172.21.15.132:0/819107883 shutdown_connections 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.728+0000 7fbef3821700 1 --2- 172.21.15.132:0/819107883 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbeec0703b0 0x7fbeec100d60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.728+0000 7fbef3821700 1 -- 172.21.15.132:0/819107883 >> 172.21.15.132:0/819107883 conn(0x7fbeec0f63c0 msgr2=0x7fbeec0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:33.893 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.728+0000 7fbef3821700 1 -- 172.21.15.132:0/819107883 shutdown_connections 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.728+0000 7fbef3821700 1 -- 172.21.15.132:0/819107883 wait complete. 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.728+0000 7fbef3821700 1 Processor -- start 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.729+0000 7fbef3821700 1 -- start start 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.729+0000 7fbef3821700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbeec0703b0 0x7fbeec10cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.729+0000 7fbef3821700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeec10d100 con 0x7fbeec0703b0 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.729+0000 7fbef15bd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbeec0703b0 0x7fbeec10cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.729+0000 7fbef15bd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbeec0703b0 0x7fbeec10cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:58482/0 (socket says 172.21.15.132:58482) 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.729+0000 7fbef15bd700 1 -- 172.21.15.132:0/3392789270 learned_addr learned my addr 172.21.15.132:0/3392789270 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.729+0000 7fbef15bd700 1 -- 172.21.15.132:0/3392789270 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbee800b200 con 0x7fbeec0703b0 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.729+0000 7fbef15bd700 1 --2- 172.21.15.132:0/3392789270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbeec0703b0 0x7fbeec10cc10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fbee800b190 tx=0x7fbee8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.729+0000 7fbed9ffb700 1 -- 172.21.15.132:0/3392789270 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbee801b410 con 0x7fbeec0703b0 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.729+0000 7fbed9ffb700 1 -- 172.21.15.132:0/3392789270 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbee801b9e0 con 0x7fbeec0703b0 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.729+0000 7fbef3821700 1 -- 172.21.15.132:0/3392789270 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeec10d300 con 0x7fbeec0703b0 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.730+0000 7fbed9ffb700 1 -- 172.21.15.132:0/3392789270 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbee801a880 con 0x7fbeec0703b0 2023-12-06T03:52:33.894 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.730+0000 7fbef3821700 1 -- 172.21.15.132:0/3392789270 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeec10d710 con 0x7fbeec0703b0 2023-12-06T03:52:33.895 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.730+0000 7fbed9ffb700 1 -- 172.21.15.132:0/3392789270 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fbee800f070 con 0x7fbeec0703b0 2023-12-06T03:52:33.895 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.731+0000 7fbed9ffb700 1 --2- 172.21.15.132:0/3392789270 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fbed402fd50 0x7fbed4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:33.895 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.731+0000 7fbed9ffb700 1 -- 172.21.15.132:0/3392789270 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fbee80412b0 con 0x7fbeec0703b0 2023-12-06T03:52:33.895 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.731+0000 7fbef3821700 1 -- 172.21.15.132:0/3392789270 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeec045ad0 con 0x7fbeec0703b0 2023-12-06T03:52:33.895 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.731+0000 7fbef0dbc700 1 --2- 172.21.15.132:0/3392789270 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fbed402fd50 0x7fbed4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:33.895 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.731+0000 7fbef0dbc700 1 --2- 172.21.15.132:0/3392789270 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fbed402fd50 0x7fbed4032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fbedc0096a0 tx=0x7fbedc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:33.895 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.735+0000 7fbed9ffb700 1 -- 172.21.15.132:0/3392789270 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbee8010360 con 0x7fbeec0703b0 2023-12-06T03:52:33.913 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.912+0000 7fbef3821700 1 -- 172.21.15.132:0/3392789270 --> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fbeec106aa0 con 0x7fbed402fd50 2023-12-06T03:52:33.917 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.916+0000 7fbed9ffb700 1 -- 172.21.15.132:0/3392789270 <== mgr.14114 v2:172.21.15.132:6800/3766660979 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fbeec106aa0 con 0x7fbed402fd50 2023-12-06T03:52:33.917 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-12-06T03:52:33.920 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.918+0000 7fbef3821700 1 -- 172.21.15.132:0/3392789270 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fbed402fd50 msgr2=0x7fbed4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:33.920 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.918+0000 7fbef3821700 1 --2- 172.21.15.132:0/3392789270 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fbed402fd50 0x7fbed4032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fbedc0096a0 tx=0x7fbedc006b40).stop 2023-12-06T03:52:33.920 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.918+0000 7fbef3821700 1 -- 172.21.15.132:0/3392789270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbeec0703b0 msgr2=0x7fbeec10cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:33.920 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.918+0000 7fbef3821700 1 --2- 172.21.15.132:0/3392789270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbeec0703b0 0x7fbeec10cc10 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fbee800b190 tx=0x7fbee8006ee0).stop 2023-12-06T03:52:33.920 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.918+0000 7fbef3821700 1 -- 172.21.15.132:0/3392789270 shutdown_connections 2023-12-06T03:52:33.920 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.918+0000 7fbef3821700 1 --2- 172.21.15.132:0/3392789270 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fbed402fd50 0x7fbed4032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:33.920 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.918+0000 7fbef3821700 1 --2- 172.21.15.132:0/3392789270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbeec0703b0 0x7fbeec10cc10 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:33.920 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.918+0000 7fbef3821700 1 -- 172.21.15.132:0/3392789270 >> 172.21.15.132:0/3392789270 conn(0x7fbeec0f63c0 msgr2=0x7fbeec0f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:33.920 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.918+0000 7fbef3821700 1 -- 172.21.15.132:0/3392789270 shutdown_connections 2023-12-06T03:52:33.920 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:33.918+0000 7fbef3821700 1 -- 172.21.15.132:0/3392789270 wait complete. 2023-12-06T03:52:35.030 INFO:teuthology.orchestra.run.smithi132.stderr:Deploying crash service with default placement... 2023-12-06T03:52:35.030 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-12-06T03:52:35.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:34 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:33.913885+0000 mgr.smithi132.gokxja (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-06T03:52:35.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:34 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:52:33.914389+0000 mgr.smithi132.gokxja (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-12-06T03:52:35.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:34 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:33.916562+0000 mon.smithi132 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:35.449 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.445+0000 7fafcc9f5700 1 Processor -- start 2023-12-06T03:52:35.449 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.445+0000 7fafcc9f5700 1 -- start start 2023-12-06T03:52:35.449 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.445+0000 7fafcc9f5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafc80fab90 0x7fafc80fafb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:35.449 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.445+0000 7fafcc9f5700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc80fb4a0 con 0x7fafc80fab90 2023-12-06T03:52:35.449 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.445+0000 7fafc659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafc80fab90 0x7fafc80fafb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:35.449 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.445+0000 7fafc659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafc80fab90 0x7fafc80fafb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:58488/0 (socket says 172.21.15.132:58488) 2023-12-06T03:52:35.449 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.445+0000 7fafc659c700 1 -- 172.21.15.132:0/3114325709 learned_addr learned my addr 172.21.15.132:0/3114325709 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:35.449 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.446+0000 7fafc659c700 1 -- 172.21.15.132:0/3114325709 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc80fb5e0 con 0x7fafc80fab90 2023-12-06T03:52:35.449 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.446+0000 7fafc659c700 1 --2- 172.21.15.132:0/3114325709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafc80fab90 0x7fafc80fafb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fafb800b870 tx=0x7fafb8006f70).ready entity=mon.0 client_cookie=e82035fd94f41404 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:35.449 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.446+0000 7fafc559a700 1 -- 172.21.15.132:0/3114325709 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafb800da20 con 0x7fafc80fab90 2023-12-06T03:52:35.450 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.446+0000 7fafc559a700 1 -- 172.21.15.132:0/3114325709 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fafb8010470 con 0x7fafc80fab90 2023-12-06T03:52:35.450 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.446+0000 7fafc559a700 1 -- 172.21.15.132:0/3114325709 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafb800f5b0 con 0x7fafc80fab90 2023-12-06T03:52:35.450 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.447+0000 7fafcc9f5700 1 -- 172.21.15.132:0/3114325709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafc80fab90 msgr2=0x7fafc80fafb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:35.450 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.447+0000 7fafcc9f5700 1 --2- 172.21.15.132:0/3114325709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafc80fab90 0x7fafc80fafb0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fafb800b870 tx=0x7fafb8006f70).stop 2023-12-06T03:52:35.450 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.447+0000 7fafcc9f5700 1 -- 172.21.15.132:0/3114325709 shutdown_connections 2023-12-06T03:52:35.450 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.447+0000 7fafcc9f5700 1 --2- 172.21.15.132:0/3114325709 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafc80fab90 0x7fafc80fafb0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:35.450 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.447+0000 7fafcc9f5700 1 -- 172.21.15.132:0/3114325709 >> 172.21.15.132:0/3114325709 conn(0x7fafc80f63b0 msgr2=0x7fafc80f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:35.450 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.447+0000 7fafcc9f5700 1 -- 172.21.15.132:0/3114325709 shutdown_connections 2023-12-06T03:52:35.450 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.447+0000 7fafcc9f5700 1 -- 172.21.15.132:0/3114325709 wait complete. 2023-12-06T03:52:35.450 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.447+0000 7fafcc9f5700 1 Processor -- start 2023-12-06T03:52:35.450 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.447+0000 7fafcc9f5700 1 -- start start 2023-12-06T03:52:35.450 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.448+0000 7fafcc9f5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafc80fab90 0x7fafc8108730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:35.450 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.448+0000 7fafcc9f5700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc8108c20 con 0x7fafc80fab90 2023-12-06T03:52:35.450 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.448+0000 7fafc659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafc80fab90 0x7fafc8108730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:35.451 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.448+0000 7fafc659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafc80fab90 0x7fafc8108730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:58494/0 (socket says 172.21.15.132:58494) 2023-12-06T03:52:35.451 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.448+0000 7fafc659c700 1 -- 172.21.15.132:0/554071507 learned_addr learned my addr 172.21.15.132:0/554071507 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:35.451 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.448+0000 7fafc659c700 1 -- 172.21.15.132:0/554071507 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb800b480 con 0x7fafc80fab90 2023-12-06T03:52:35.451 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.448+0000 7fafc659c700 1 --2- 172.21.15.132:0/554071507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafc80fab90 0x7fafc8108730 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fafb8006a40 tx=0x7fafb8006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:35.451 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.448+0000 7fafb6ffd700 1 -- 172.21.15.132:0/554071507 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafb800f3d0 con 0x7fafc80fab90 2023-12-06T03:52:35.451 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.448+0000 7fafcc9f5700 1 -- 172.21.15.132:0/554071507 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc8108e20 con 0x7fafc80fab90 2023-12-06T03:52:35.451 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.448+0000 7fafb6ffd700 1 -- 172.21.15.132:0/554071507 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fafb800fef0 con 0x7fafc80fab90 2023-12-06T03:52:35.452 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.448+0000 7fafb6ffd700 1 -- 172.21.15.132:0/554071507 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafb8019d40 con 0x7fafc80fab90 2023-12-06T03:52:35.452 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.448+0000 7fafcc9f5700 1 -- 172.21.15.132:0/554071507 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc8109230 con 0x7fafc80fab90 2023-12-06T03:52:35.452 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.449+0000 7fafb6ffd700 1 -- 172.21.15.132:0/554071507 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fafb802b060 con 0x7fafc80fab90 2023-12-06T03:52:35.452 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.449+0000 7fafb6ffd700 1 --2- 172.21.15.132:0/554071507 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fafac02fd50 0x7fafac032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:35.452 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.449+0000 7fafcc9f5700 1 -- 172.21.15.132:0/554071507 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc8045ad0 con 0x7fafc80fab90 2023-12-06T03:52:35.452 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.449+0000 7fafb6ffd700 1 -- 172.21.15.132:0/554071507 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fafb8044120 con 0x7fafc80fab90 2023-12-06T03:52:35.452 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.449+0000 7fafc5d9b700 1 --2- 172.21.15.132:0/554071507 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fafac02fd50 0x7fafac032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:35.452 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.450+0000 7fafc5d9b700 1 --2- 172.21.15.132:0/554071507 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fafac02fd50 0x7fafac032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fafbc0096a0 tx=0x7fafbc006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:35.454 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.453+0000 7fafb6ffd700 1 -- 172.21.15.132:0/554071507 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafb8014070 con 0x7fafc80fab90 2023-12-06T03:52:35.640 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.638+0000 7fafcc9f5700 1 -- 172.21.15.132:0/554071507 --> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fafc8109b10 con 0x7fafac02fd50 2023-12-06T03:52:35.643 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.642+0000 7fafb6ffd700 1 -- 172.21.15.132:0/554071507 <== mgr.14114 v2:172.21.15.132:6800/3766660979 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fafc8109b10 con 0x7fafac02fd50 2023-12-06T03:52:35.643 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: Scheduled crash update... 2023-12-06T03:52:35.645 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.644+0000 7fafcc9f5700 1 -- 172.21.15.132:0/554071507 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fafac02fd50 msgr2=0x7fafac032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:35.645 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.644+0000 7fafcc9f5700 1 --2- 172.21.15.132:0/554071507 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fafac02fd50 0x7fafac032210 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fafbc0096a0 tx=0x7fafbc006b40).stop 2023-12-06T03:52:35.646 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.644+0000 7fafcc9f5700 1 -- 172.21.15.132:0/554071507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafc80fab90 msgr2=0x7fafc8108730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:35.646 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.644+0000 7fafcc9f5700 1 --2- 172.21.15.132:0/554071507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafc80fab90 0x7fafc8108730 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fafb8006a40 tx=0x7fafb8006f70).stop 2023-12-06T03:52:35.646 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.644+0000 7fafcc9f5700 1 -- 172.21.15.132:0/554071507 shutdown_connections 2023-12-06T03:52:35.646 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.644+0000 7fafcc9f5700 1 --2- 172.21.15.132:0/554071507 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fafac02fd50 0x7fafac032210 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:35.646 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.644+0000 7fafcc9f5700 1 --2- 172.21.15.132:0/554071507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fafc80fab90 0x7fafc8108730 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:35.646 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.644+0000 7fafcc9f5700 1 -- 172.21.15.132:0/554071507 >> 172.21.15.132:0/554071507 conn(0x7fafc80f63b0 msgr2=0x7fafc80ff920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:35.646 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.644+0000 7fafcc9f5700 1 -- 172.21.15.132:0/554071507 shutdown_connections 2023-12-06T03:52:35.646 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:35.644+0000 7fafcc9f5700 1 -- 172.21.15.132:0/554071507 wait complete. 2023-12-06T03:52:36.156 INFO:teuthology.orchestra.run.smithi132.stderr:Enabling mgr prometheus module... 2023-12-06T03:52:36.156 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-12-06T03:52:36.506 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.504+0000 7f35cb59e700 1 Processor -- start 2023-12-06T03:52:36.506 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.504+0000 7f35cb59e700 1 -- start start 2023-12-06T03:52:36.506 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.504+0000 7f35cb59e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35cc0fa9d0 0x7f35cc0fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:36.506 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.504+0000 7f35cb59e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc0fb2e0 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.507 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.504+0000 7f35ca59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35cc0fa9d0 0x7f35cc0fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:36.507 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.504+0000 7f35ca59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35cc0fa9d0 0x7f35cc0fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:58508/0 (socket says 172.21.15.132:58508) 2023-12-06T03:52:36.507 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.504+0000 7f35ca59c700 1 -- 172.21.15.132:0/3661669294 learned_addr learned my addr 172.21.15.132:0/3661669294 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:36.507 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.505+0000 7f35ca59c700 1 -- 172.21.15.132:0/3661669294 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35cc0fb420 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.507 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.505+0000 7f35ca59c700 1 --2- 172.21.15.132:0/3661669294 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35cc0fa9d0 0x7f35cc0fadf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f35bc00b5f0 tx=0x7f35bc006ee0).ready entity=mon.0 client_cookie=3cb3a5ed79d740b7 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:36.507 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.505+0000 7f35c959a700 1 -- 172.21.15.132:0/3661669294 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35bc00d600 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.507 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.505+0000 7f35c959a700 1 -- 172.21.15.132:0/3661669294 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35bc00dc10 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.507 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.505+0000 7f35c959a700 1 -- 172.21.15.132:0/3661669294 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35bc011cd0 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.507 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.506+0000 7f35cb59e700 1 -- 172.21.15.132:0/3661669294 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35cc0fa9d0 msgr2=0x7f35cc0fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:36.507 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.506+0000 7f35cb59e700 1 --2- 172.21.15.132:0/3661669294 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35cc0fa9d0 0x7f35cc0fadf0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f35bc00b5f0 tx=0x7f35bc006ee0).stop 2023-12-06T03:52:36.510 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.506+0000 7f35cb59e700 1 -- 172.21.15.132:0/3661669294 shutdown_connections 2023-12-06T03:52:36.510 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.506+0000 7f35cb59e700 1 --2- 172.21.15.132:0/3661669294 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35cc0fa9d0 0x7f35cc0fadf0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:36.510 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.506+0000 7f35cb59e700 1 -- 172.21.15.132:0/3661669294 >> 172.21.15.132:0/3661669294 conn(0x7f35cc0f6210 msgr2=0x7f35cc0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:36.510 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.506+0000 7f35cb59e700 1 -- 172.21.15.132:0/3661669294 shutdown_connections 2023-12-06T03:52:36.510 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.506+0000 7f35cb59e700 1 -- 172.21.15.132:0/3661669294 wait complete. 2023-12-06T03:52:36.510 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.506+0000 7f35cb59e700 1 Processor -- start 2023-12-06T03:52:36.510 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.506+0000 7f35cb59e700 1 -- start start 2023-12-06T03:52:36.510 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.506+0000 7f35cb59e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35cc0fa9d0 0x7f35cc10fa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:36.510 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.506+0000 7f35cb59e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35cc10b3f0 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.510 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.507+0000 7f35ca59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35cc0fa9d0 0x7f35cc10fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:36.510 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.507+0000 7f35ca59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35cc0fa9d0 0x7f35cc10fa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:58514/0 (socket says 172.21.15.132:58514) 2023-12-06T03:52:36.510 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.507+0000 7f35ca59c700 1 -- 172.21.15.132:0/757785360 learned_addr learned my addr 172.21.15.132:0/757785360 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:36.511 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.507+0000 7f35ca59c700 1 -- 172.21.15.132:0/757785360 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35bc00b200 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.511 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.507+0000 7f35ca59c700 1 --2- 172.21.15.132:0/757785360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35cc0fa9d0 0x7f35cc10fa00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f35bc00dfb0 tx=0x7f35bc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:36.511 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.507+0000 7f35baffd700 1 -- 172.21.15.132:0/757785360 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35bc011df0 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.511 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.507+0000 7f35baffd700 1 -- 172.21.15.132:0/757785360 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35bc01b8c0 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.511 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.507+0000 7f35cb59e700 1 -- 172.21.15.132:0/757785360 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35cc070450 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.511 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.507+0000 7f35baffd700 1 -- 172.21.15.132:0/757785360 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35bc01a8c0 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.511 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.507+0000 7f35cb59e700 1 -- 172.21.15.132:0/757785360 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35cc10fef0 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.511 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.508+0000 7f35cb59e700 1 -- 172.21.15.132:0/757785360 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35cc045ad0 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.511 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.508+0000 7f35baffd700 1 -- 172.21.15.132:0/757785360 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 7) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f35bc00f070 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.511 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.509+0000 7f35baffd700 1 --2- 172.21.15.132:0/757785360 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f35b002fdb0 0x7f35b0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:36.511 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.509+0000 7f35baffd700 1 -- 172.21.15.132:0/757785360 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f35bc040eb0 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.511 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.509+0000 7f35c9d9b700 1 --2- 172.21.15.132:0/757785360 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f35b002fdb0 0x7f35b0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:36.511 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.509+0000 7f35c9d9b700 1 --2- 172.21.15.132:0/757785360 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f35b002fdb0 0x7f35b0032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f35c00096a0 tx=0x7f35c0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:36.518 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.516+0000 7f35baffd700 1 -- 172.21.15.132:0/757785360 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35bc019d60 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.719 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:36.718+0000 7f35cb59e700 1 -- 172.21.15.132:0/757785360 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f35cc027a00 con 0x7f35cc0fa9d0 2023-12-06T03:52:36.972 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:36 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:35.639567+0000 mgr.smithi132.gokxja (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-12-06T03:52:36.972 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:36 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:52:35.640191+0000 mgr.smithi132.gokxja (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2023-12-06T03:52:36.972 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:36 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:35.642387+0000 mon.smithi132 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:36.972 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:36 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:36.273058+0000 mon.smithi132 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:36.972 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:36 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:36.498902+0000 mon.smithi132 (mon.0) 59 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:37.647 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:37.646+0000 7f35baffd700 1 -- 172.21.15.132:0/757785360 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f35bc01b410 con 0x7f35cc0fa9d0 2023-12-06T03:52:37.648 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:37.647+0000 7f35baffd700 1 -- 172.21.15.132:0/757785360 <== mon.0 v2:172.21.15.132:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v8) v1 ==== 89+0+0 (secure 0 0 0) 0x7f35bc045bb0 con 0x7f35cc0fa9d0 2023-12-06T03:52:37.652 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:37.650+0000 7f35cb59e700 1 -- 172.21.15.132:0/757785360 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f35b002fdb0 msgr2=0x7f35b0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:37.652 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:37.650+0000 7f35cb59e700 1 --2- 172.21.15.132:0/757785360 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f35b002fdb0 0x7f35b0032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f35c00096a0 tx=0x7f35c0006b40).stop 2023-12-06T03:52:37.652 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:37.650+0000 7f35cb59e700 1 -- 172.21.15.132:0/757785360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35cc0fa9d0 msgr2=0x7f35cc10fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:37.652 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:37.650+0000 7f35cb59e700 1 --2- 172.21.15.132:0/757785360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35cc0fa9d0 0x7f35cc10fa00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f35bc00dfb0 tx=0x7f35bc006ee0).stop 2023-12-06T03:52:37.652 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:37.650+0000 7f35cb59e700 1 -- 172.21.15.132:0/757785360 shutdown_connections 2023-12-06T03:52:37.653 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:37.650+0000 7f35cb59e700 1 --2- 172.21.15.132:0/757785360 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7f35b002fdb0 0x7f35b0032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:37.653 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:37.650+0000 7f35cb59e700 1 --2- 172.21.15.132:0/757785360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35cc0fa9d0 0x7f35cc10fa00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:37.653 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:37.650+0000 7f35cb59e700 1 -- 172.21.15.132:0/757785360 >> 172.21.15.132:0/757785360 conn(0x7f35cc0f6210 msgr2=0x7f35cc0f7c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:37.653 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:37.650+0000 7f35cb59e700 1 -- 172.21.15.132:0/757785360 shutdown_connections 2023-12-06T03:52:37.653 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:37.650+0000 7f35cb59e700 1 -- 172.21.15.132:0/757785360 wait complete. 2023-12-06T03:52:38.020 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:36.719510+0000 mon.smithi132 (mon.0) 60 : audit [INF] from='client.? 172.21.15.132:0/757785360' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-12-06T03:52:38.020 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:37.115553+0000 mon.smithi132 (mon.0) 61 : audit [INF] from='mgr.14114 172.21.15.132:0/3235991157' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:38.220 INFO:teuthology.orchestra.run.smithi132.stderr:Deploying prometheus service with default placement... 2023-12-06T03:52:38.221 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-12-06T03:52:38.604 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.602+0000 7fb7f0a9a700 1 Processor -- start 2023-12-06T03:52:38.604 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.602+0000 7fb7f0a9a700 1 -- start start 2023-12-06T03:52:38.604 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.602+0000 7fb7f0a9a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7ec0fab80 0x7fb7ec0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:38.605 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.602+0000 7fb7f0a9a700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec0fb490 con 0x7fb7ec0fab80 2023-12-06T03:52:38.605 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.602+0000 7fb7ea59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7ec0fab80 0x7fb7ec0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:38.605 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.602+0000 7fb7ea59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7ec0fab80 0x7fb7ec0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59452/0 (socket says 172.21.15.132:59452) 2023-12-06T03:52:38.605 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.602+0000 7fb7ea59c700 1 -- 172.21.15.132:0/2488034598 learned_addr learned my addr 172.21.15.132:0/2488034598 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:38.605 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.603+0000 7fb7ea59c700 1 -- 172.21.15.132:0/2488034598 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7ec0fb5d0 con 0x7fb7ec0fab80 2023-12-06T03:52:38.605 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.603+0000 7fb7ea59c700 1 --2- 172.21.15.132:0/2488034598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7ec0fab80 0x7fb7ec0fafa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fb7dc00b870 tx=0x7fb7dc006f70).ready entity=mon.0 client_cookie=a1aeaf7ea04f291a server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:38.605 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.603+0000 7fb7e959a700 1 -- 172.21.15.132:0/2488034598 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7dc00da20 con 0x7fb7ec0fab80 2023-12-06T03:52:38.605 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.603+0000 7fb7e959a700 1 -- 172.21.15.132:0/2488034598 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7dc010470 con 0x7fb7ec0fab80 2023-12-06T03:52:38.605 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.603+0000 7fb7e959a700 1 -- 172.21.15.132:0/2488034598 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7dc00f5b0 con 0x7fb7ec0fab80 2023-12-06T03:52:38.608 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.604+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/2488034598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7ec0fab80 msgr2=0x7fb7ec0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:38.608 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.604+0000 7fb7f0a9a700 1 --2- 172.21.15.132:0/2488034598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7ec0fab80 0x7fb7ec0fafa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fb7dc00b870 tx=0x7fb7dc006f70).stop 2023-12-06T03:52:38.608 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.604+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/2488034598 shutdown_connections 2023-12-06T03:52:38.608 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.604+0000 7fb7f0a9a700 1 --2- 172.21.15.132:0/2488034598 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7ec0fab80 0x7fb7ec0fafa0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:38.608 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.604+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/2488034598 >> 172.21.15.132:0/2488034598 conn(0x7fb7ec0f63c0 msgr2=0x7fb7ec0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:38.608 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.604+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/2488034598 shutdown_connections 2023-12-06T03:52:38.608 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.604+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/2488034598 wait complete. 2023-12-06T03:52:38.608 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.605+0000 7fb7f0a9a700 1 Processor -- start 2023-12-06T03:52:38.608 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.605+0000 7fb7f0a9a700 1 -- start start 2023-12-06T03:52:38.608 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.605+0000 7fb7f0a9a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7ec0fab80 0x7fb7ec10ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.605+0000 7fb7f0a9a700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7ec10d1c0 con 0x7fb7ec0fab80 2023-12-06T03:52:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.605+0000 7fb7ea59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7ec0fab80 0x7fb7ec10ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.605+0000 7fb7ea59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7ec0fab80 0x7fb7ec10ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59466/0 (socket says 172.21.15.132:59466) 2023-12-06T03:52:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.605+0000 7fb7ea59c700 1 -- 172.21.15.132:0/1166017757 learned_addr learned my addr 172.21.15.132:0/1166017757 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.605+0000 7fb7ea59c700 1 -- 172.21.15.132:0/1166017757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc00b480 con 0x7fb7ec0fab80 2023-12-06T03:52:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.605+0000 7fb7ea59c700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7ec0fab80 0x7fb7ec10ccd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb7dc00b840 tx=0x7fb7dc006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.606+0000 7fb7daffd700 1 -- 172.21.15.132:0/1166017757 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7dc00de60 con 0x7fb7ec0fab80 2023-12-06T03:52:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.606+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/1166017757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7ec10d3c0 con 0x7fb7ec0fab80 2023-12-06T03:52:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.606+0000 7fb7daffd700 1 -- 172.21.15.132:0/1166017757 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7dc00fef0 con 0x7fb7ec0fab80 2023-12-06T03:52:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.606+0000 7fb7daffd700 1 -- 172.21.15.132:0/1166017757 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7dc019de0 con 0x7fb7ec0fab80 2023-12-06T03:52:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.606+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/1166017757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7ec10d7d0 con 0x7fb7ec0fab80 2023-12-06T03:52:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.607+0000 7fb7daffd700 1 -- 172.21.15.132:0/1166017757 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 8) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fb7dc016030 con 0x7fb7ec0fab80 2023-12-06T03:52:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.607+0000 7fb7daffd700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fb7d002fdb0 0x7fb7d0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:38.609 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.607+0000 7fb7daffd700 1 -- 172.21.15.132:0/1166017757 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fb7dc041e50 con 0x7fb7ec0fab80 2023-12-06T03:52:38.610 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.607+0000 7fb7e9d9b700 1 -- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fb7d002fdb0 msgr2=0x7fb7d0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/3766660979 2023-12-06T03:52:38.610 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.607+0000 7fb7e9d9b700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fb7d002fdb0 0x7fb7d0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-06T03:52:38.610 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.607+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/1166017757 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7ec0fc900 con 0x7fb7ec0fab80 2023-12-06T03:52:38.612 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.611+0000 7fb7daffd700 1 -- 172.21.15.132:0/1166017757 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7dc014020 con 0x7fb7ec0fab80 2023-12-06T03:52:38.806 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.805+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/1166017757 --> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7ec0fcac0 con 0x7fb7d002fdb0 2023-12-06T03:52:38.808 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.808+0000 7fb7e9d9b700 1 -- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fb7d002fdb0 msgr2=0x7fb7d0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/3766660979 2023-12-06T03:52:38.809 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:38.808+0000 7fb7e9d9b700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fb7d002fdb0 0x7fb7d0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-06T03:52:39.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:38 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:37.648145+0000 mon.smithi132 (mon.0) 62 : audit [INF] from='client.? 172.21.15.132:0/757785360' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-12-06T03:52:39.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:38 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:37.648189+0000 mon.smithi132 (mon.0) 63 : cluster [DBG] mgrmap e8: smithi132.gokxja(active, since 16s) 2023-12-06T03:52:39.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:39.209+0000 7fb7e9d9b700 1 -- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fb7d002fdb0 msgr2=0x7fb7d0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/3766660979 2023-12-06T03:52:39.210 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:39.209+0000 7fb7e9d9b700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fb7d002fdb0 0x7fb7d0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-06T03:52:40.011 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:40.009+0000 7fb7e9d9b700 1 -- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fb7d002fdb0 msgr2=0x7fb7d0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/3766660979 2023-12-06T03:52:40.011 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:40.009+0000 7fb7e9d9b700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fb7d002fdb0 0x7fb7d0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-06T03:52:41.612 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:41.610+0000 7fb7e9d9b700 1 -- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fb7d002fdb0 msgr2=0x7fb7d0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/3766660979 2023-12-06T03:52:41.612 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:41.610+0000 7fb7e9d9b700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fb7d002fdb0 0x7fb7d0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-06T03:52:42.616 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:42.613+0000 7fb7daffd700 1 -- 172.21.15.132:0/1166017757 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 9) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fb7dc02cc90 con 0x7fb7ec0fab80 2023-12-06T03:52:42.616 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:42.613+0000 7fb7daffd700 1 -- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fb7d002fdb0 msgr2=0x7fb7d0032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-06T03:52:42.616 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:42.613+0000 7fb7daffd700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fb7d002fdb0 0x7fb7d0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-06T03:52:43.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:42 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:42.563184+0000 mon.smithi132 (mon.0) 64 : cluster [INF] Active manager daemon smithi132.gokxja restarted 2023-12-06T03:52:43.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:42 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:42.563558+0000 mon.smithi132 (mon.0) 65 : cluster [INF] Activating manager daemon smithi132.gokxja 2023-12-06T03:52:43.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:42 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:42.565151+0000 mon.smithi132 (mon.0) 66 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-12-06T03:52:43.620 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.617+0000 7fb7daffd700 1 -- 172.21.15.132:0/1166017757 <== mon.0 v2:172.21.15.132:3300/0 8 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fb7dc044d30 con 0x7fb7ec0fab80 2023-12-06T03:52:43.620 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.618+0000 7fb7daffd700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fb7d0033c20 0x7fb7d0036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:43.621 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.618+0000 7fb7daffd700 1 -- 172.21.15.132:0/1166017757 --> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fb7ec0fcac0 con 0x7fb7d0033c20 2023-12-06T03:52:43.621 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.618+0000 7fb7e9d9b700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fb7d0033c20 0x7fb7d0036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:43.621 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.618+0000 7fb7e9d9b700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fb7d0033c20 0x7fb7d0036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb7e00036d0 tx=0x7fb7e0006cb0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:43.632 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.630+0000 7fb7daffd700 1 -- 172.21.15.132:0/1166017757 <== mgr.14142 v2:172.21.15.132:6800/597241079 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fb7ec0fcac0 con 0x7fb7d0033c20 2023-12-06T03:52:43.632 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-12-06T03:52:43.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.632+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fb7d0033c20 msgr2=0x7fb7d0036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:43.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.632+0000 7fb7f0a9a700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fb7d0033c20 0x7fb7d0036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb7e00036d0 tx=0x7fb7e0006cb0).stop 2023-12-06T03:52:43.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.632+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7ec0fab80 msgr2=0x7fb7ec10ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:43.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.632+0000 7fb7f0a9a700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7ec0fab80 0x7fb7ec10ccd0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb7dc00b840 tx=0x7fb7dc006f70).stop 2023-12-06T03:52:43.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.632+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/1166017757 shutdown_connections 2023-12-06T03:52:43.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.632+0000 7fb7f0a9a700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fb7d0033c20 0x7fb7d0036010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:43.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.632+0000 7fb7f0a9a700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb7ec0fab80 0x7fb7ec10ccd0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:43.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.632+0000 7fb7f0a9a700 1 --2- 172.21.15.132:0/1166017757 >> [v2:172.21.15.132:6800/3766660979,v1:172.21.15.132:6801/3766660979] conn(0x7fb7d002fdb0 0x7fb7d0032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-06T03:52:43.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.632+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/1166017757 >> 172.21.15.132:0/1166017757 conn(0x7fb7ec0f63c0 msgr2=0x7fb7ec0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:43.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.632+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/1166017757 shutdown_connections 2023-12-06T03:52:43.635 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:43.632+0000 7fb7f0a9a700 1 -- 172.21.15.132:0/1166017757 wait complete. 2023-12-06T03:52:43.810 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:42.615949+0000 mon.smithi132 (mon.0) 67 : cluster [DBG] mgrmap e9: smithi132.gokxja(active, starting, since 0.0524763s) 2023-12-06T03:52:43.811 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:42.616932+0000 mon.smithi132 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.132:0/2348744226' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mon metadata", "id": "smithi132"}]: dispatch 2023-12-06T03:52:43.811 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:42.618944+0000 mon.smithi132 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.132:0/2348744226' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mgr metadata", "who": "smithi132.gokxja", "id": "smithi132.gokxja"}]: dispatch 2023-12-06T03:52:43.812 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:42.619163+0000 mon.smithi132 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.132:0/2348744226' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-06T03:52:43.812 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:42.619306+0000 mon.smithi132 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.132:0/2348744226' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-06T03:52:43.812 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:42.619447+0000 mon.smithi132 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.132:0/2348744226' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-06T03:52:43.812 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:42.632983+0000 mon.smithi132 (mon.0) 73 : cluster [INF] Manager daemon smithi132.gokxja is now available 2023-12-06T03:52:43.812 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:42.733585+0000 mon.smithi132 (mon.0) 74 : audit [DBG] from='mgr.14142 172.21.15.132:0/2348744226' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-06T03:52:43.812 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:42.734956+0000 mon.smithi132 (mon.0) 75 : audit [DBG] from='mgr.14142 172.21.15.132:0/2348744226' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-06T03:52:43.812 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:42.750288+0000 mon.smithi132 (mon.0) 76 : audit [INF] from='mgr.14142 172.21.15.132:0/2348744226' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/mirror_snapshot_schedule"}]: dispatch 2023-12-06T03:52:43.812 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:42.758644+0000 mon.smithi132 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.132:0/2348744226' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/trash_purge_schedule"}]: dispatch 2023-12-06T03:52:43.936 INFO:teuthology.orchestra.run.smithi132.stderr:Deploying grafana service with default placement... 2023-12-06T03:52:43.936 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-12-06T03:52:44.366 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.360+0000 7fe99caa9700 1 Processor -- start 2023-12-06T03:52:44.366 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.360+0000 7fe99caa9700 1 -- start start 2023-12-06T03:52:44.366 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.360+0000 7fe99caa9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9980703f0 0x7fe9980fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:44.367 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.360+0000 7fe99caa9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9980fee10 con 0x7fe9980703f0 2023-12-06T03:52:44.367 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.360+0000 7fe99659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9980703f0 0x7fe9980fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:44.367 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.360+0000 7fe99659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9980703f0 0x7fe9980fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59534/0 (socket says 172.21.15.132:59534) 2023-12-06T03:52:44.367 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.360+0000 7fe99659c700 1 -- 172.21.15.132:0/1634505626 learned_addr learned my addr 172.21.15.132:0/1634505626 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:44.367 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.361+0000 7fe99659c700 1 -- 172.21.15.132:0/1634505626 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9980fef50 con 0x7fe9980703f0 2023-12-06T03:52:44.367 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.361+0000 7fe99659c700 1 --2- 172.21.15.132:0/1634505626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9980703f0 0x7fe9980fe920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fe98800b5f0 tx=0x7fe988006ee0).ready entity=mon.0 client_cookie=819026f5b79d33e8 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:44.367 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.361+0000 7fe99559a700 1 -- 172.21.15.132:0/1634505626 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe98800d600 con 0x7fe9980703f0 2023-12-06T03:52:44.367 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.361+0000 7fe99559a700 1 -- 172.21.15.132:0/1634505626 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe98800dc10 con 0x7fe9980703f0 2023-12-06T03:52:44.367 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.361+0000 7fe99559a700 1 -- 172.21.15.132:0/1634505626 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe988011cd0 con 0x7fe9980703f0 2023-12-06T03:52:44.367 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.361+0000 7fe99caa9700 1 -- 172.21.15.132:0/1634505626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9980703f0 msgr2=0x7fe9980fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:44.367 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.361+0000 7fe99caa9700 1 --2- 172.21.15.132:0/1634505626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9980703f0 0x7fe9980fe920 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fe98800b5f0 tx=0x7fe988006ee0).stop 2023-12-06T03:52:44.367 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.362+0000 7fe99caa9700 1 -- 172.21.15.132:0/1634505626 shutdown_connections 2023-12-06T03:52:44.367 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.362+0000 7fe99caa9700 1 --2- 172.21.15.132:0/1634505626 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9980703f0 0x7fe9980fe920 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:44.367 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.362+0000 7fe99caa9700 1 -- 172.21.15.132:0/1634505626 >> 172.21.15.132:0/1634505626 conn(0x7fe9980f63c0 msgr2=0x7fe9980f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.362+0000 7fe99caa9700 1 -- 172.21.15.132:0/1634505626 shutdown_connections 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.362+0000 7fe99caa9700 1 -- 172.21.15.132:0/1634505626 wait complete. 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.362+0000 7fe99caa9700 1 Processor -- start 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.362+0000 7fe99caa9700 1 -- start start 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.362+0000 7fe99caa9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9980703f0 0x7fe9980fc3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.362+0000 7fe99caa9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9980fa820 con 0x7fe9980703f0 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.363+0000 7fe99659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9980703f0 0x7fe9980fc3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.363+0000 7fe99659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9980703f0 0x7fe9980fc3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59536/0 (socket says 172.21.15.132:59536) 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.363+0000 7fe99659c700 1 -- 172.21.15.132:0/3976422323 learned_addr learned my addr 172.21.15.132:0/3976422323 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.363+0000 7fe99659c700 1 -- 172.21.15.132:0/3976422323 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe98800b200 con 0x7fe9980703f0 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.363+0000 7fe99659c700 1 --2- 172.21.15.132:0/3976422323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9980703f0 0x7fe9980fc3a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe98800df90 tx=0x7fe988006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.363+0000 7fe986ffd700 1 -- 172.21.15.132:0/3976422323 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe988011da0 con 0x7fe9980703f0 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.363+0000 7fe986ffd700 1 -- 172.21.15.132:0/3976422323 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe98801b8c0 con 0x7fe9980703f0 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.363+0000 7fe99caa9700 1 -- 172.21.15.132:0/3976422323 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9980fa9c0 con 0x7fe9980703f0 2023-12-06T03:52:44.368 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.363+0000 7fe986ffd700 1 -- 172.21.15.132:0/3976422323 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe98801a930 con 0x7fe9980703f0 2023-12-06T03:52:44.369 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.363+0000 7fe99caa9700 1 -- 172.21.15.132:0/3976422323 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9980fadd0 con 0x7fe9980703f0 2023-12-06T03:52:44.369 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.364+0000 7fe986ffd700 1 -- 172.21.15.132:0/3976422323 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fe98800f070 con 0x7fe9980703f0 2023-12-06T03:52:44.369 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.364+0000 7fe986ffd700 1 --2- 172.21.15.132:0/3976422323 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fe97c02fd10 0x7fe97c0321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:44.369 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.364+0000 7fe986ffd700 1 -- 172.21.15.132:0/3976422323 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fe988040e90 con 0x7fe9980703f0 2023-12-06T03:52:44.369 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.364+0000 7fe99caa9700 1 -- 172.21.15.132:0/3976422323 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe998045ad0 con 0x7fe9980703f0 2023-12-06T03:52:44.369 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.364+0000 7fe995d9b700 1 --2- 172.21.15.132:0/3976422323 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fe97c02fd10 0x7fe97c0321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:44.369 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.365+0000 7fe995d9b700 1 --2- 172.21.15.132:0/3976422323 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fe97c02fd10 0x7fe97c0321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe98c0096a0 tx=0x7fe98c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:44.372 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.370+0000 7fe986ffd700 1 -- 172.21.15.132:0/3976422323 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe988016070 con 0x7fe9980703f0 2023-12-06T03:52:44.556 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.554+0000 7fe99caa9700 1 -- 172.21.15.132:0/3976422323 --> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fe998104900 con 0x7fe97c02fd10 2023-12-06T03:52:44.560 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.558+0000 7fe986ffd700 1 -- 172.21.15.132:0/3976422323 <== mgr.14142 v2:172.21.15.132:6800/597241079 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fe998104900 con 0x7fe97c02fd10 2023-12-06T03:52:44.560 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-12-06T03:52:44.564 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.561+0000 7fe99caa9700 1 -- 172.21.15.132:0/3976422323 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fe97c02fd10 msgr2=0x7fe97c0321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:44.564 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.561+0000 7fe99caa9700 1 --2- 172.21.15.132:0/3976422323 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fe97c02fd10 0x7fe97c0321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fe98c0096a0 tx=0x7fe98c006b40).stop 2023-12-06T03:52:44.564 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.561+0000 7fe99caa9700 1 -- 172.21.15.132:0/3976422323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9980703f0 msgr2=0x7fe9980fc3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:44.564 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.561+0000 7fe99caa9700 1 --2- 172.21.15.132:0/3976422323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9980703f0 0x7fe9980fc3a0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe98800df90 tx=0x7fe988006ee0).stop 2023-12-06T03:52:44.564 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.561+0000 7fe99caa9700 1 -- 172.21.15.132:0/3976422323 shutdown_connections 2023-12-06T03:52:44.564 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.561+0000 7fe99caa9700 1 --2- 172.21.15.132:0/3976422323 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fe97c02fd10 0x7fe97c0321d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:44.565 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.561+0000 7fe99caa9700 1 --2- 172.21.15.132:0/3976422323 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9980703f0 0x7fe9980fc3a0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:44.565 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.561+0000 7fe99caa9700 1 -- 172.21.15.132:0/3976422323 >> 172.21.15.132:0/3976422323 conn(0x7fe9980f63c0 msgr2=0x7fe9980f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:44.565 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.561+0000 7fe99caa9700 1 -- 172.21.15.132:0/3976422323 shutdown_connections 2023-12-06T03:52:44.565 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:44.561+0000 7fe99caa9700 1 -- 172.21.15.132:0/3976422323 wait complete. 2023-12-06T03:52:44.738 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:44 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:43.621033+0000 mon.smithi132 (mon.0) 78 : cluster [DBG] mgrmap e10: smithi132.gokxja(active, since 1.05756s) 2023-12-06T03:52:44.738 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:44 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:52:43.622289+0000 mgr.smithi132.gokxja (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-12-06T03:52:44.739 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:44 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:43.631578+0000 mon.smithi132 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.132:0/2348744226' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:44.739 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:44 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:43.645892+0000 mon.smithi132 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.132:0/2348744226' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:44.739 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:44 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:44.559521+0000 mon.smithi132 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.132:0/2348744226' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:45.078 INFO:teuthology.orchestra.run.smithi132.stderr:Deploying node-exporter service with default placement... 2023-12-06T03:52:45.079 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-12-06T03:52:45.461 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.456+0000 7f76cc71e700 1 Processor -- start 2023-12-06T03:52:45.461 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.456+0000 7f76cc71e700 1 -- start start 2023-12-06T03:52:45.461 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.456+0000 7f76cc71e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76c40fab80 0x7f76c40fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:45.461 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.456+0000 7f76cc71e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c40fb490 con 0x7f76c40fab80 2023-12-06T03:52:45.461 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.457+0000 7f76ca4ba700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76c40fab80 0x7f76c40fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:45.461 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.457+0000 7f76ca4ba700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76c40fab80 0x7f76c40fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59540/0 (socket says 172.21.15.132:59540) 2023-12-06T03:52:45.461 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.457+0000 7f76ca4ba700 1 -- 172.21.15.132:0/1694563515 learned_addr learned my addr 172.21.15.132:0/1694563515 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:45.461 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.457+0000 7f76ca4ba700 1 -- 172.21.15.132:0/1694563515 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c40fb5d0 con 0x7f76c40fab80 2023-12-06T03:52:45.461 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.457+0000 7f76ca4ba700 1 --2- 172.21.15.132:0/1694563515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76c40fab80 0x7f76c40fafa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f76c000b5f0 tx=0x7f76c0006ee0).ready entity=mon.0 client_cookie=dcacf30c4a6119f6 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:45.461 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.457+0000 7f76c94b8700 1 -- 172.21.15.132:0/1694563515 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76c000d600 con 0x7f76c40fab80 2023-12-06T03:52:45.462 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.457+0000 7f76c94b8700 1 -- 172.21.15.132:0/1694563515 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76c000dc10 con 0x7f76c40fab80 2023-12-06T03:52:45.462 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.457+0000 7f76c94b8700 1 -- 172.21.15.132:0/1694563515 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76c0011cd0 con 0x7f76c40fab80 2023-12-06T03:52:45.462 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.458+0000 7f76cc71e700 1 -- 172.21.15.132:0/1694563515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76c40fab80 msgr2=0x7f76c40fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:45.462 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.458+0000 7f76cc71e700 1 --2- 172.21.15.132:0/1694563515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76c40fab80 0x7f76c40fafa0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f76c000b5f0 tx=0x7f76c0006ee0).stop 2023-12-06T03:52:45.462 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.458+0000 7f76cc71e700 1 -- 172.21.15.132:0/1694563515 shutdown_connections 2023-12-06T03:52:45.462 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.458+0000 7f76cc71e700 1 --2- 172.21.15.132:0/1694563515 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76c40fab80 0x7f76c40fafa0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:45.462 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.458+0000 7f76cc71e700 1 -- 172.21.15.132:0/1694563515 >> 172.21.15.132:0/1694563515 conn(0x7f76c40f63c0 msgr2=0x7f76c40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:45.462 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.458+0000 7f76cc71e700 1 -- 172.21.15.132:0/1694563515 shutdown_connections 2023-12-06T03:52:45.462 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.458+0000 7f76cc71e700 1 -- 172.21.15.132:0/1694563515 wait complete. 2023-12-06T03:52:45.462 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.458+0000 7f76cc71e700 1 Processor -- start 2023-12-06T03:52:45.462 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.458+0000 7f76cc71e700 1 -- start start 2023-12-06T03:52:45.462 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.459+0000 7f76cc71e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76c40fab80 0x7f76c410cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:45.462 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.459+0000 7f76cc71e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c410d1d0 con 0x7f76c40fab80 2023-12-06T03:52:45.462 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.459+0000 7f76ca4ba700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76c40fab80 0x7f76c410cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:45.463 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.459+0000 7f76ca4ba700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76c40fab80 0x7f76c410cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59556/0 (socket says 172.21.15.132:59556) 2023-12-06T03:52:45.463 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.459+0000 7f76ca4ba700 1 -- 172.21.15.132:0/3097285108 learned_addr learned my addr 172.21.15.132:0/3097285108 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:45.463 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.459+0000 7f76ca4ba700 1 -- 172.21.15.132:0/3097285108 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c000b200 con 0x7f76c40fab80 2023-12-06T03:52:45.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.459+0000 7f76ca4ba700 1 --2- 172.21.15.132:0/3097285108 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76c40fab80 0x7f76c410cce0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f76c000e040 tx=0x7f76c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:45.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.459+0000 7f76b2ffd700 1 -- 172.21.15.132:0/3097285108 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76c001b410 con 0x7f76c40fab80 2023-12-06T03:52:45.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.459+0000 7f76b2ffd700 1 -- 172.21.15.132:0/3097285108 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76c001ba20 con 0x7f76c40fab80 2023-12-06T03:52:45.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.459+0000 7f76cc71e700 1 -- 172.21.15.132:0/3097285108 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c410d3d0 con 0x7f76c40fab80 2023-12-06T03:52:45.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.459+0000 7f76b2ffd700 1 -- 172.21.15.132:0/3097285108 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76c001a910 con 0x7f76c40fab80 2023-12-06T03:52:45.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.459+0000 7f76cc71e700 1 -- 172.21.15.132:0/3097285108 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c410d7e0 con 0x7f76c40fab80 2023-12-06T03:52:45.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.460+0000 7f76b2ffd700 1 -- 172.21.15.132:0/3097285108 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 10) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f76c000f070 con 0x7f76c40fab80 2023-12-06T03:52:45.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.460+0000 7f76b2ffd700 1 --2- 172.21.15.132:0/3097285108 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f76ac02fd40 0x7f76ac032200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:45.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.460+0000 7f76cc71e700 1 -- 172.21.15.132:0/3097285108 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c4045ad0 con 0x7f76c40fab80 2023-12-06T03:52:45.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.460+0000 7f76b2ffd700 1 -- 172.21.15.132:0/3097285108 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f76c0041020 con 0x7f76c40fab80 2023-12-06T03:52:45.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.460+0000 7f76c9cb9700 1 --2- 172.21.15.132:0/3097285108 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f76ac02fd40 0x7f76ac032200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:45.464 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.461+0000 7f76c9cb9700 1 --2- 172.21.15.132:0/3097285108 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f76ac02fd40 0x7f76ac032200 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f76b40096a0 tx=0x7f76b4006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:45.466 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.464+0000 7f76b2ffd700 1 -- 172.21.15.132:0/3097285108 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76c0016070 con 0x7f76c40fab80 2023-12-06T03:52:45.560 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.558+0000 7f76b2ffd700 1 -- 172.21.15.132:0/3097285108 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f76c0024790 con 0x7f76c40fab80 2023-12-06T03:52:45.645 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.643+0000 7f76cc71e700 1 -- 172.21.15.132:0/3097285108 --> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f76c410e040 con 0x7f76ac02fd40 2023-12-06T03:52:45.651 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.649+0000 7f76b2ffd700 1 -- 172.21.15.132:0/3097285108 <== mgr.14142 v2:172.21.15.132:6800/597241079 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f76c410e040 con 0x7f76ac02fd40 2023-12-06T03:52:45.651 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-12-06T03:52:45.654 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.651+0000 7f76cc71e700 1 -- 172.21.15.132:0/3097285108 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f76ac02fd40 msgr2=0x7f76ac032200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:45.654 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.651+0000 7f76cc71e700 1 --2- 172.21.15.132:0/3097285108 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f76ac02fd40 0x7f76ac032200 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f76b40096a0 tx=0x7f76b4006b40).stop 2023-12-06T03:52:45.654 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.651+0000 7f76cc71e700 1 -- 172.21.15.132:0/3097285108 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76c40fab80 msgr2=0x7f76c410cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:45.654 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.651+0000 7f76cc71e700 1 --2- 172.21.15.132:0/3097285108 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76c40fab80 0x7f76c410cce0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f76c000e040 tx=0x7f76c0006ee0).stop 2023-12-06T03:52:45.654 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.651+0000 7f76cc71e700 1 -- 172.21.15.132:0/3097285108 shutdown_connections 2023-12-06T03:52:45.654 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.651+0000 7f76cc71e700 1 --2- 172.21.15.132:0/3097285108 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f76ac02fd40 0x7f76ac032200 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:45.654 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.651+0000 7f76cc71e700 1 --2- 172.21.15.132:0/3097285108 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76c40fab80 0x7f76c410cce0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:45.655 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.651+0000 7f76cc71e700 1 -- 172.21.15.132:0/3097285108 >> 172.21.15.132:0/3097285108 conn(0x7f76c40f63c0 msgr2=0x7f76c4103d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:45.655 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.652+0000 7f76cc71e700 1 -- 172.21.15.132:0/3097285108 shutdown_connections 2023-12-06T03:52:45.655 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:45.652+0000 7f76cc71e700 1 -- 172.21.15.132:0/3097285108 wait complete. 2023-12-06T03:52:45.980 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:45 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:44.556624+0000 mgr.smithi132.gokxja (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-12-06T03:52:45.980 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:45 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:52:44.557142+0000 mgr.smithi132.gokxja (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-12-06T03:52:45.980 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:45 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:45.560724+0000 mon.smithi132 (mon.0) 82 : cluster [DBG] mgrmap e11: smithi132.gokxja(active, since 2s) 2023-12-06T03:52:46.028 INFO:teuthology.orchestra.run.smithi132.stderr:Deploying alertmanager service with default placement... 2023-12-06T03:52:46.028 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-12-06T03:52:46.385 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.381+0000 7f8944232700 1 Processor -- start 2023-12-06T03:52:46.386 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.381+0000 7f8944232700 1 -- start start 2023-12-06T03:52:46.386 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.381+0000 7f8944232700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c0fab80 0x7f893c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:46.386 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.381+0000 7f8944232700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c0fb490 con 0x7f893c0fab80 2023-12-06T03:52:46.386 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.382+0000 7f8941fce700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c0fab80 0x7f893c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:46.386 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.382+0000 7f8941fce700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c0fab80 0x7f893c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59566/0 (socket says 172.21.15.132:59566) 2023-12-06T03:52:46.386 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.382+0000 7f8941fce700 1 -- 172.21.15.132:0/2022375539 learned_addr learned my addr 172.21.15.132:0/2022375539 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:46.386 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.382+0000 7f8941fce700 1 -- 172.21.15.132:0/2022375539 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f893c0fb5d0 con 0x7f893c0fab80 2023-12-06T03:52:46.386 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.382+0000 7f8941fce700 1 --2- 172.21.15.132:0/2022375539 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c0fab80 0x7f893c0fafa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f892c00b870 tx=0x7f892c006f70).ready entity=mon.0 client_cookie=ece3035b245a4e6d server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:46.386 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.382+0000 7f8940fcc700 1 -- 172.21.15.132:0/2022375539 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f892c00da20 con 0x7f893c0fab80 2023-12-06T03:52:46.386 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.382+0000 7f8940fcc700 1 -- 172.21.15.132:0/2022375539 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f892c010470 con 0x7f893c0fab80 2023-12-06T03:52:46.386 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.382+0000 7f8940fcc700 1 -- 172.21.15.132:0/2022375539 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f892c00f5b0 con 0x7f893c0fab80 2023-12-06T03:52:46.386 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.383+0000 7f8944232700 1 -- 172.21.15.132:0/2022375539 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c0fab80 msgr2=0x7f893c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:46.387 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.383+0000 7f8944232700 1 --2- 172.21.15.132:0/2022375539 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c0fab80 0x7f893c0fafa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f892c00b870 tx=0x7f892c006f70).stop 2023-12-06T03:52:46.387 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.383+0000 7f8944232700 1 -- 172.21.15.132:0/2022375539 shutdown_connections 2023-12-06T03:52:46.387 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.383+0000 7f8944232700 1 --2- 172.21.15.132:0/2022375539 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c0fab80 0x7f893c0fafa0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:46.387 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.383+0000 7f8944232700 1 -- 172.21.15.132:0/2022375539 >> 172.21.15.132:0/2022375539 conn(0x7f893c0f63c0 msgr2=0x7f893c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:46.387 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.383+0000 7f8944232700 1 -- 172.21.15.132:0/2022375539 shutdown_connections 2023-12-06T03:52:46.387 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.383+0000 7f8944232700 1 -- 172.21.15.132:0/2022375539 wait complete. 2023-12-06T03:52:46.387 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.383+0000 7f8944232700 1 Processor -- start 2023-12-06T03:52:46.387 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.383+0000 7f8944232700 1 -- start start 2023-12-06T03:52:46.387 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.383+0000 7f8944232700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c0fab80 0x7f893c108820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:46.389 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.383+0000 7f8944232700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f893c108d10 con 0x7f893c0fab80 2023-12-06T03:52:46.389 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.384+0000 7f8941fce700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c0fab80 0x7f893c108820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:46.389 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.384+0000 7f8941fce700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c0fab80 0x7f893c108820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59580/0 (socket says 172.21.15.132:59580) 2023-12-06T03:52:46.389 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.384+0000 7f8941fce700 1 -- 172.21.15.132:0/2602357307 learned_addr learned my addr 172.21.15.132:0/2602357307 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:46.389 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.384+0000 7f8941fce700 1 -- 172.21.15.132:0/2602357307 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892c00b480 con 0x7f893c0fab80 2023-12-06T03:52:46.389 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.384+0000 7f8941fce700 1 --2- 172.21.15.132:0/2602357307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c0fab80 0x7f893c108820 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f892c006800 tx=0x7f892c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:46.389 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.384+0000 7f892a7fc700 1 -- 172.21.15.132:0/2602357307 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f892c00f3d0 con 0x7f893c0fab80 2023-12-06T03:52:46.390 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.384+0000 7f8944232700 1 -- 172.21.15.132:0/2602357307 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f893c108f10 con 0x7f893c0fab80 2023-12-06T03:52:46.390 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.384+0000 7f892a7fc700 1 -- 172.21.15.132:0/2602357307 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f892c00fea0 con 0x7f893c0fab80 2023-12-06T03:52:46.390 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.384+0000 7f892a7fc700 1 -- 172.21.15.132:0/2602357307 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f892c019d10 con 0x7f893c0fab80 2023-12-06T03:52:46.390 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.384+0000 7f8944232700 1 -- 172.21.15.132:0/2602357307 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f893c109320 con 0x7f893c0fab80 2023-12-06T03:52:46.390 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.385+0000 7f892a7fc700 1 -- 172.21.15.132:0/2602357307 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f892c016030 con 0x7f893c0fab80 2023-12-06T03:52:46.390 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.385+0000 7f892a7fc700 1 --2- 172.21.15.132:0/2602357307 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f892402feb0 0x7f8924032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:46.390 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.385+0000 7f892a7fc700 1 -- 172.21.15.132:0/2602357307 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f892c0454c0 con 0x7f893c0fab80 2023-12-06T03:52:46.390 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.385+0000 7f89417cd700 1 --2- 172.21.15.132:0/2602357307 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f892402feb0 0x7f8924032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:46.390 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.385+0000 7f8944232700 1 -- 172.21.15.132:0/2602357307 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f893c059100 con 0x7f893c0fab80 2023-12-06T03:52:46.390 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.385+0000 7f89417cd700 1 --2- 172.21.15.132:0/2602357307 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f892402feb0 0x7f8924032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f89380096a0 tx=0x7f8938006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:46.391 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.389+0000 7f892a7fc700 1 -- 172.21.15.132:0/2602357307 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f892c014070 con 0x7f893c0fab80 2023-12-06T03:52:46.570 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.568+0000 7f8944232700 1 -- 172.21.15.132:0/2602357307 --> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f893c102640 con 0x7f892402feb0 2023-12-06T03:52:46.573 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.571+0000 7f892a7fc700 1 -- 172.21.15.132:0/2602357307 <== mgr.14142 v2:172.21.15.132:6800/597241079 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f893c102640 con 0x7f892402feb0 2023-12-06T03:52:46.574 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-12-06T03:52:46.577 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.573+0000 7f8944232700 1 -- 172.21.15.132:0/2602357307 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f892402feb0 msgr2=0x7f8924032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:46.577 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.573+0000 7f8944232700 1 --2- 172.21.15.132:0/2602357307 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f892402feb0 0x7f8924032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f89380096a0 tx=0x7f8938006b40).stop 2023-12-06T03:52:46.577 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.573+0000 7f8944232700 1 -- 172.21.15.132:0/2602357307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c0fab80 msgr2=0x7f893c108820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:46.577 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.573+0000 7f8944232700 1 --2- 172.21.15.132:0/2602357307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c0fab80 0x7f893c108820 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f892c006800 tx=0x7f892c006f70).stop 2023-12-06T03:52:46.577 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.573+0000 7f8944232700 1 -- 172.21.15.132:0/2602357307 shutdown_connections 2023-12-06T03:52:46.577 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.573+0000 7f8944232700 1 --2- 172.21.15.132:0/2602357307 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f892402feb0 0x7f8924032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:46.577 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.573+0000 7f8944232700 1 --2- 172.21.15.132:0/2602357307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f893c0fab80 0x7f893c108820 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:46.578 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.573+0000 7f8944232700 1 -- 172.21.15.132:0/2602357307 >> 172.21.15.132:0/2602357307 conn(0x7f893c0f63c0 msgr2=0x7f893c0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:46.578 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.573+0000 7f8944232700 1 -- 172.21.15.132:0/2602357307 shutdown_connections 2023-12-06T03:52:46.578 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:46.573+0000 7f8944232700 1 -- 172.21.15.132:0/2602357307 wait complete. 2023-12-06T03:52:46.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:46 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:45.645720+0000 mgr.smithi132.gokxja (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-12-06T03:52:46.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:46 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:52:45.646443+0000 mgr.smithi132.gokxja (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-12-06T03:52:46.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:46 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:45.650894+0000 mon.smithi132 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.132:0/2348744226' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:46.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:46 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:46.572844+0000 mon.smithi132 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.132:0/2348744226' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:46.936 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-12-06T03:52:47.283 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.278+0000 7f51798dc700 1 Processor -- start 2023-12-06T03:52:47.283 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.279+0000 7f51798dc700 1 -- start start 2023-12-06T03:52:47.283 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.279+0000 7f51798dc700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51740703d0 0x7f5174100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:47.283 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.279+0000 7f51798dc700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51740fc4b0 con 0x7f51740703d0 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.279+0000 7f51788da700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51740703d0 0x7f5174100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.279+0000 7f51788da700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51740703d0 0x7f5174100ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59582/0 (socket says 172.21.15.132:59582) 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.279+0000 7f51788da700 1 -- 172.21.15.132:0/143627649 learned_addr learned my addr 172.21.15.132:0/143627649 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.279+0000 7f51788da700 1 -- 172.21.15.132:0/143627649 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5174100fb0 con 0x7f51740703d0 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.279+0000 7f51788da700 1 --2- 172.21.15.132:0/143627649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51740703d0 0x7f5174100ac0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f516800b5f0 tx=0x7f5168006ee0).ready entity=mon.0 client_cookie=34702b2c8e84f973 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.279+0000 7f51737fe700 1 -- 172.21.15.132:0/143627649 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f516800d600 con 0x7f51740703d0 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.280+0000 7f51737fe700 1 -- 172.21.15.132:0/143627649 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f516800dc10 con 0x7f51740703d0 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.280+0000 7f51737fe700 1 -- 172.21.15.132:0/143627649 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5168011cd0 con 0x7f51740703d0 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.280+0000 7f51798dc700 1 -- 172.21.15.132:0/143627649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51740703d0 msgr2=0x7f5174100ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.280+0000 7f51798dc700 1 --2- 172.21.15.132:0/143627649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51740703d0 0x7f5174100ac0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f516800b5f0 tx=0x7f5168006ee0).stop 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.280+0000 7f51798dc700 1 -- 172.21.15.132:0/143627649 shutdown_connections 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.280+0000 7f51798dc700 1 --2- 172.21.15.132:0/143627649 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51740703d0 0x7f5174100ac0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.280+0000 7f51798dc700 1 -- 172.21.15.132:0/143627649 >> 172.21.15.132:0/143627649 conn(0x7f51740f6100 msgr2=0x7f51740f8520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.280+0000 7f51798dc700 1 -- 172.21.15.132:0/143627649 shutdown_connections 2023-12-06T03:52:47.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.280+0000 7f51798dc700 1 -- 172.21.15.132:0/143627649 wait complete. 2023-12-06T03:52:47.285 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.281+0000 7f51798dc700 1 Processor -- start 2023-12-06T03:52:47.285 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.281+0000 7f51798dc700 1 -- start start 2023-12-06T03:52:47.285 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.281+0000 7f51798dc700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51740703d0 0x7f517410ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:47.285 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.281+0000 7f51798dc700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f517410cf80 con 0x7f51740703d0 2023-12-06T03:52:47.285 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.281+0000 7f51788da700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51740703d0 0x7f517410ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:47.285 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.281+0000 7f51788da700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51740703d0 0x7f517410ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:59594/0 (socket says 172.21.15.132:59594) 2023-12-06T03:52:47.285 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.281+0000 7f51788da700 1 -- 172.21.15.132:0/2525206599 learned_addr learned my addr 172.21.15.132:0/2525206599 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:47.286 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.281+0000 7f51788da700 1 -- 172.21.15.132:0/2525206599 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516800b200 con 0x7f51740703d0 2023-12-06T03:52:47.286 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.281+0000 7f51788da700 1 --2- 172.21.15.132:0/2525206599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51740703d0 0x7f517410ca90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f516800bf40 tx=0x7f5168006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:47.286 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.281+0000 7f51717fa700 1 -- 172.21.15.132:0/2525206599 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5168011c90 con 0x7f51740703d0 2023-12-06T03:52:47.286 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.282+0000 7f51717fa700 1 -- 172.21.15.132:0/2525206599 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f516801b8c0 con 0x7f51740703d0 2023-12-06T03:52:47.286 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.282+0000 7f51798dc700 1 -- 172.21.15.132:0/2525206599 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f517410d180 con 0x7f51740703d0 2023-12-06T03:52:47.286 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.282+0000 7f51717fa700 1 -- 172.21.15.132:0/2525206599 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f516801a8e0 con 0x7f51740703d0 2023-12-06T03:52:47.286 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.282+0000 7f51798dc700 1 -- 172.21.15.132:0/2525206599 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f517410d610 con 0x7f51740703d0 2023-12-06T03:52:47.286 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.282+0000 7f51717fa700 1 -- 172.21.15.132:0/2525206599 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f516800f070 con 0x7f51740703d0 2023-12-06T03:52:47.287 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.282+0000 7f51798dc700 1 -- 172.21.15.132:0/2525206599 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5174045ad0 con 0x7f51740703d0 2023-12-06T03:52:47.287 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.282+0000 7f51717fa700 1 --2- 172.21.15.132:0/2525206599 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f515802fe60 0x7f5158032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:47.287 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.283+0000 7f51717fa700 1 -- 172.21.15.132:0/2525206599 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f51680409b0 con 0x7f51740703d0 2023-12-06T03:52:47.287 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.283+0000 7f5173fff700 1 --2- 172.21.15.132:0/2525206599 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f515802fe60 0x7f5158032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:47.287 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.283+0000 7f5173fff700 1 --2- 172.21.15.132:0/2525206599 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f515802fe60 0x7f5158032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f51600096a0 tx=0x7f5160006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:47.289 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.287+0000 7f51717fa700 1 -- 172.21.15.132:0/2525206599 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f51680198f0 con 0x7f51740703d0 2023-12-06T03:52:47.441 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.439+0000 7f51798dc700 1 -- 172.21.15.132:0/2525206599 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f5174101c40 con 0x7f51740703d0 2023-12-06T03:52:47.444 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.442+0000 7f51717fa700 1 -- 172.21.15.132:0/2525206599 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f5168016070 con 0x7f51740703d0 2023-12-06T03:52:47.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.444+0000 7f51798dc700 1 -- 172.21.15.132:0/2525206599 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f515802fe60 msgr2=0x7f5158032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:47.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.444+0000 7f51798dc700 1 --2- 172.21.15.132:0/2525206599 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f515802fe60 0x7f5158032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f51600096a0 tx=0x7f5160006b40).stop 2023-12-06T03:52:47.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.444+0000 7f51798dc700 1 -- 172.21.15.132:0/2525206599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51740703d0 msgr2=0x7f517410ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:47.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.444+0000 7f51798dc700 1 --2- 172.21.15.132:0/2525206599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51740703d0 0x7f517410ca90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f516800bf40 tx=0x7f5168006ee0).stop 2023-12-06T03:52:47.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.444+0000 7f51798dc700 1 -- 172.21.15.132:0/2525206599 shutdown_connections 2023-12-06T03:52:47.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.444+0000 7f51798dc700 1 --2- 172.21.15.132:0/2525206599 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f515802fe60 0x7f5158032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:47.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.444+0000 7f51798dc700 1 --2- 172.21.15.132:0/2525206599 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51740703d0 0x7f517410ca90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:47.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.444+0000 7f51798dc700 1 -- 172.21.15.132:0/2525206599 >> 172.21.15.132:0/2525206599 conn(0x7f51740f6100 msgr2=0x7f51740f7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:47.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.444+0000 7f51798dc700 1 -- 172.21.15.132:0/2525206599 shutdown_connections 2023-12-06T03:52:47.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:47.444+0000 7f51798dc700 1 -- 172.21.15.132:0/2525206599 wait complete. 2023-12-06T03:52:47.779 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:47 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:46.570117+0000 mgr.smithi132.gokxja (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-12-06T03:52:47.780 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:47 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:52:46.570659+0000 mgr.smithi132.gokxja (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-12-06T03:52:47.780 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:47 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:47.443920+0000 mon.smithi132 (mon.0) 85 : audit [INF] from='client.? 172.21.15.132:0/2525206599' entity='client.admin' 2023-12-06T03:52:47.819 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-12-06T03:52:48.152 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.149+0000 7f0abe09d700 1 Processor -- start 2023-12-06T03:52:48.152 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.149+0000 7f0abe09d700 1 -- start start 2023-12-06T03:52:48.152 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.149+0000 7f0abe09d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ab8070430 0x7f0ab80fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:48.152 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.149+0000 7f0abe09d700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab80fedd0 con 0x7f0ab8070430 2023-12-06T03:52:48.153 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.149+0000 7f0ab77fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ab8070430 0x7f0ab80fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:48.153 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.150+0000 7f0ab77fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ab8070430 0x7f0ab80fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33244/0 (socket says 172.21.15.132:33244) 2023-12-06T03:52:48.153 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.150+0000 7f0ab77fe700 1 -- 172.21.15.132:0/822321851 learned_addr learned my addr 172.21.15.132:0/822321851 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:48.153 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.150+0000 7f0ab77fe700 1 -- 172.21.15.132:0/822321851 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ab80fef10 con 0x7f0ab8070430 2023-12-06T03:52:48.153 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.150+0000 7f0ab77fe700 1 --2- 172.21.15.132:0/822321851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ab8070430 0x7f0ab80fe8e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f0aac00b5f0 tx=0x7f0aac006ee0).ready entity=mon.0 client_cookie=e0456844d86ee97b server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:48.154 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.150+0000 7f0ab67fc700 1 -- 172.21.15.132:0/822321851 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aac00d600 con 0x7f0ab8070430 2023-12-06T03:52:48.154 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.150+0000 7f0ab67fc700 1 -- 172.21.15.132:0/822321851 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0aac00dc10 con 0x7f0ab8070430 2023-12-06T03:52:48.154 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.150+0000 7f0ab67fc700 1 -- 172.21.15.132:0/822321851 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aac011cd0 con 0x7f0ab8070430 2023-12-06T03:52:48.154 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.151+0000 7f0abe09d700 1 -- 172.21.15.132:0/822321851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ab8070430 msgr2=0x7f0ab80fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:48.154 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.151+0000 7f0abe09d700 1 --2- 172.21.15.132:0/822321851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ab8070430 0x7f0ab80fe8e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f0aac00b5f0 tx=0x7f0aac006ee0).stop 2023-12-06T03:52:48.154 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.151+0000 7f0abe09d700 1 -- 172.21.15.132:0/822321851 shutdown_connections 2023-12-06T03:52:48.154 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.151+0000 7f0abe09d700 1 --2- 172.21.15.132:0/822321851 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ab8070430 0x7f0ab80fe8e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:48.154 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.151+0000 7f0abe09d700 1 -- 172.21.15.132:0/822321851 >> 172.21.15.132:0/822321851 conn(0x7f0ab80f63c0 msgr2=0x7f0ab80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:48.154 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.151+0000 7f0abe09d700 1 -- 172.21.15.132:0/822321851 shutdown_connections 2023-12-06T03:52:48.154 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.151+0000 7f0abe09d700 1 -- 172.21.15.132:0/822321851 wait complete. 2023-12-06T03:52:48.155 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.151+0000 7f0abe09d700 1 Processor -- start 2023-12-06T03:52:48.155 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.151+0000 7f0abe09d700 1 -- start start 2023-12-06T03:52:48.155 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.152+0000 7f0abe09d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ab8070430 0x7f0ab810ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:48.155 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.152+0000 7f0abe09d700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab810b020 con 0x7f0ab8070430 2023-12-06T03:52:48.155 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.152+0000 7f0ab77fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ab8070430 0x7f0ab810ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:48.155 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.152+0000 7f0ab77fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ab8070430 0x7f0ab810ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33252/0 (socket says 172.21.15.132:33252) 2023-12-06T03:52:48.155 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.152+0000 7f0ab77fe700 1 -- 172.21.15.132:0/4045072900 learned_addr learned my addr 172.21.15.132:0/4045072900 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:48.155 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.152+0000 7f0ab77fe700 1 -- 172.21.15.132:0/4045072900 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac00b200 con 0x7f0ab8070430 2023-12-06T03:52:48.156 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.152+0000 7f0ab77fe700 1 --2- 172.21.15.132:0/4045072900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ab8070430 0x7f0ab810ab30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f0aac00dfe0 tx=0x7f0aac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:48.156 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.152+0000 7f0a9ffff700 1 -- 172.21.15.132:0/4045072900 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aac011db0 con 0x7f0ab8070430 2023-12-06T03:52:48.156 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.152+0000 7f0a9ffff700 1 -- 172.21.15.132:0/4045072900 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0aac01b8c0 con 0x7f0ab8070430 2023-12-06T03:52:48.156 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.152+0000 7f0abe09d700 1 -- 172.21.15.132:0/4045072900 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab810b220 con 0x7f0ab8070430 2023-12-06T03:52:48.156 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.152+0000 7f0a9ffff700 1 -- 172.21.15.132:0/4045072900 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aac01a840 con 0x7f0ab8070430 2023-12-06T03:52:48.156 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.152+0000 7f0abe09d700 1 -- 172.21.15.132:0/4045072900 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab810b630 con 0x7f0ab8070430 2023-12-06T03:52:48.156 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.153+0000 7f0a9ffff700 1 -- 172.21.15.132:0/4045072900 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f0aac00f070 con 0x7f0ab8070430 2023-12-06T03:52:48.156 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.153+0000 7f0a9ffff700 1 --2- 172.21.15.132:0/4045072900 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f0aa402fe60 0x7f0aa4032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:48.156 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.153+0000 7f0abe09d700 1 -- 172.21.15.132:0/4045072900 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab8045ad0 con 0x7f0ab8070430 2023-12-06T03:52:48.157 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.153+0000 7f0a9ffff700 1 -- 172.21.15.132:0/4045072900 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0aac041150 con 0x7f0ab8070430 2023-12-06T03:52:48.157 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.153+0000 7f0ab6ffd700 1 --2- 172.21.15.132:0/4045072900 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f0aa402fe60 0x7f0aa4032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:48.157 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.154+0000 7f0ab6ffd700 1 --2- 172.21.15.132:0/4045072900 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f0aa402fe60 0x7f0aa4032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f0aa80096a0 tx=0x7f0aa8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:48.159 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.157+0000 7f0a9ffff700 1 -- 172.21.15.132:0/4045072900 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0aac0107d0 con 0x7f0ab8070430 2023-12-06T03:52:48.308 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.306+0000 7f0abe09d700 1 -- 172.21.15.132:0/4045072900 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f0ab80ffc00 con 0x7f0ab8070430 2023-12-06T03:52:48.310 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.309+0000 7f0a9ffff700 1 -- 172.21.15.132:0/4045072900 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7f0aac022e40 con 0x7f0ab8070430 2023-12-06T03:52:48.313 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.310+0000 7f0abe09d700 1 -- 172.21.15.132:0/4045072900 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f0aa402fe60 msgr2=0x7f0aa4032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:48.313 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.310+0000 7f0abe09d700 1 --2- 172.21.15.132:0/4045072900 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f0aa402fe60 0x7f0aa4032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f0aa80096a0 tx=0x7f0aa8006b40).stop 2023-12-06T03:52:48.313 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.311+0000 7f0abe09d700 1 -- 172.21.15.132:0/4045072900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ab8070430 msgr2=0x7f0ab810ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:48.313 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.311+0000 7f0abe09d700 1 --2- 172.21.15.132:0/4045072900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ab8070430 0x7f0ab810ab30 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f0aac00dfe0 tx=0x7f0aac006ee0).stop 2023-12-06T03:52:48.314 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.311+0000 7f0abe09d700 1 -- 172.21.15.132:0/4045072900 shutdown_connections 2023-12-06T03:52:48.314 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.311+0000 7f0abe09d700 1 --2- 172.21.15.132:0/4045072900 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7f0aa402fe60 0x7f0aa4032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:48.314 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.311+0000 7f0abe09d700 1 --2- 172.21.15.132:0/4045072900 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0ab8070430 0x7f0ab810ab30 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:48.314 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.311+0000 7f0abe09d700 1 -- 172.21.15.132:0/4045072900 >> 172.21.15.132:0/4045072900 conn(0x7f0ab80f63c0 msgr2=0x7f0ab80f7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:48.314 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.311+0000 7f0abe09d700 1 -- 172.21.15.132:0/4045072900 shutdown_connections 2023-12-06T03:52:48.314 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.311+0000 7f0abe09d700 1 -- 172.21.15.132:0/4045072900 wait complete. 2023-12-06T03:52:48.653 INFO:teuthology.orchestra.run.smithi132.stderr:Enabling the dashboard module... 2023-12-06T03:52:48.653 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-12-06T03:52:48.998 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.993+0000 7fa18249c700 1 Processor -- start 2023-12-06T03:52:48.998 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.993+0000 7fa18249c700 1 -- start start 2023-12-06T03:52:48.998 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.994+0000 7fa18249c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa17c0fab80 0x7fa17c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.994+0000 7fa18249c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c0fb490 con 0x7fa17c0fab80 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.994+0000 7fa17bfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa17c0fab80 0x7fa17c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.994+0000 7fa17bfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa17c0fab80 0x7fa17c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33254/0 (socket says 172.21.15.132:33254) 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.994+0000 7fa17bfff700 1 -- 172.21.15.132:0/4256973835 learned_addr learned my addr 172.21.15.132:0/4256973835 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.994+0000 7fa17bfff700 1 -- 172.21.15.132:0/4256973835 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17c0fb5d0 con 0x7fa17c0fab80 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.994+0000 7fa17bfff700 1 --2- 172.21.15.132:0/4256973835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa17c0fab80 0x7fa17c0fafa0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fa17000b870 tx=0x7fa170006f70).ready entity=mon.0 client_cookie=41e8f938c4e13e09 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.994+0000 7fa17affd700 1 -- 172.21.15.132:0/4256973835 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa17000da20 con 0x7fa17c0fab80 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.994+0000 7fa17affd700 1 -- 172.21.15.132:0/4256973835 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa170010470 con 0x7fa17c0fab80 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.994+0000 7fa17affd700 1 -- 172.21.15.132:0/4256973835 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa17000f5b0 con 0x7fa17c0fab80 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.995+0000 7fa18249c700 1 -- 172.21.15.132:0/4256973835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa17c0fab80 msgr2=0x7fa17c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.995+0000 7fa18249c700 1 --2- 172.21.15.132:0/4256973835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa17c0fab80 0x7fa17c0fafa0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fa17000b870 tx=0x7fa170006f70).stop 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.995+0000 7fa18249c700 1 -- 172.21.15.132:0/4256973835 shutdown_connections 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.995+0000 7fa18249c700 1 --2- 172.21.15.132:0/4256973835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa17c0fab80 0x7fa17c0fafa0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.995+0000 7fa18249c700 1 -- 172.21.15.132:0/4256973835 >> 172.21.15.132:0/4256973835 conn(0x7fa17c0f63c0 msgr2=0x7fa17c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:48.999 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.995+0000 7fa18249c700 1 -- 172.21.15.132:0/4256973835 shutdown_connections 2023-12-06T03:52:49.000 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.995+0000 7fa18249c700 1 -- 172.21.15.132:0/4256973835 wait complete. 2023-12-06T03:52:49.000 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.996+0000 7fa18249c700 1 Processor -- start 2023-12-06T03:52:49.000 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.996+0000 7fa18249c700 1 -- start start 2023-12-06T03:52:49.000 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.996+0000 7fa18249c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa17c0fab80 0x7fa17c1087f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:49.000 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.996+0000 7fa18249c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17c108ce0 con 0x7fa17c0fab80 2023-12-06T03:52:49.000 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.996+0000 7fa17bfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa17c0fab80 0x7fa17c1087f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:49.000 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.996+0000 7fa17bfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa17c0fab80 0x7fa17c1087f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33260/0 (socket says 172.21.15.132:33260) 2023-12-06T03:52:49.000 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.996+0000 7fa17bfff700 1 -- 172.21.15.132:0/1174028425 learned_addr learned my addr 172.21.15.132:0/1174028425 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:49.000 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.996+0000 7fa17bfff700 1 -- 172.21.15.132:0/1174028425 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17000b480 con 0x7fa17c0fab80 2023-12-06T03:52:49.000 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.996+0000 7fa17bfff700 1 --2- 172.21.15.132:0/1174028425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa17c0fab80 0x7fa17c1087f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fa17000b840 tx=0x7fa170006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:49.001 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.997+0000 7fa178ff9700 1 -- 172.21.15.132:0/1174028425 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa17000f3d0 con 0x7fa17c0fab80 2023-12-06T03:52:49.001 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.997+0000 7fa178ff9700 1 -- 172.21.15.132:0/1174028425 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa17000fef0 con 0x7fa17c0fab80 2023-12-06T03:52:49.001 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.997+0000 7fa18249c700 1 -- 172.21.15.132:0/1174028425 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17c108ee0 con 0x7fa17c0fab80 2023-12-06T03:52:49.001 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.997+0000 7fa178ff9700 1 -- 172.21.15.132:0/1174028425 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa170019c70 con 0x7fa17c0fab80 2023-12-06T03:52:49.001 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.997+0000 7fa18249c700 1 -- 172.21.15.132:0/1174028425 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17c1092f0 con 0x7fa17c0fab80 2023-12-06T03:52:49.001 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.997+0000 7fa178ff9700 1 -- 172.21.15.132:0/1174028425 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 11) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fa170016030 con 0x7fa17c0fab80 2023-12-06T03:52:49.001 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.998+0000 7fa178ff9700 1 --2- 172.21.15.132:0/1174028425 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa16802feb0 0x7fa168032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:49.001 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.998+0000 7fa178ff9700 1 -- 172.21.15.132:0/1174028425 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fa170044630 con 0x7fa17c0fab80 2023-12-06T03:52:49.001 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.998+0000 7fa17b7fe700 1 --2- 172.21.15.132:0/1174028425 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa16802feb0 0x7fa168032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:49.001 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.998+0000 7fa17b7fe700 1 --2- 172.21.15.132:0/1174028425 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa16802feb0 0x7fa168032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa16c0096a0 tx=0x7fa16c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:49.001 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:48.998+0000 7fa18249c700 1 -- 172.21.15.132:0/1174028425 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa17c059100 con 0x7fa17c0fab80 2023-12-06T03:52:49.004 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:49.002+0000 7fa178ff9700 1 -- 172.21.15.132:0/1174028425 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa170014070 con 0x7fa17c0fab80 2023-12-06T03:52:49.190 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:49.188+0000 7fa18249c700 1 -- 172.21.15.132:0/1174028425 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fa17c045ad0 con 0x7fa17c0fab80 2023-12-06T03:52:49.316 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:49.314+0000 7fa178ff9700 1 -- 172.21.15.132:0/1174028425 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fa170034070 con 0x7fa17c0fab80 2023-12-06T03:52:49.317 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:49.315+0000 7fa178ff9700 1 -- 172.21.15.132:0/1174028425 <== mon.0 v2:172.21.15.132:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v12) v1 ==== 88+0+0 (secure 0 0 0) 0x7fa170024420 con 0x7fa17c0fab80 2023-12-06T03:52:49.320 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:49.318+0000 7fa18249c700 1 -- 172.21.15.132:0/1174028425 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa16802feb0 msgr2=0x7fa168032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:49.321 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:49.318+0000 7fa18249c700 1 --2- 172.21.15.132:0/1174028425 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa16802feb0 0x7fa168032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fa16c0096a0 tx=0x7fa16c006b40).stop 2023-12-06T03:52:49.321 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:49.318+0000 7fa18249c700 1 -- 172.21.15.132:0/1174028425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa17c0fab80 msgr2=0x7fa17c1087f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:49.321 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:49.318+0000 7fa18249c700 1 --2- 172.21.15.132:0/1174028425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa17c0fab80 0x7fa17c1087f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fa17000b840 tx=0x7fa170006f70).stop 2023-12-06T03:52:49.321 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:49.318+0000 7fa18249c700 1 -- 172.21.15.132:0/1174028425 shutdown_connections 2023-12-06T03:52:49.321 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:49.318+0000 7fa18249c700 1 --2- 172.21.15.132:0/1174028425 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa16802feb0 0x7fa168032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:49.321 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:49.318+0000 7fa18249c700 1 --2- 172.21.15.132:0/1174028425 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa17c0fab80 0x7fa17c1087f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:49.321 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:49.318+0000 7fa18249c700 1 -- 172.21.15.132:0/1174028425 >> 172.21.15.132:0/1174028425 conn(0x7fa17c0f63c0 msgr2=0x7fa17c0ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:49.321 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:49.318+0000 7fa18249c700 1 -- 172.21.15.132:0/1174028425 shutdown_connections 2023-12-06T03:52:49.321 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:49.318+0000 7fa18249c700 1 -- 172.21.15.132:0/1174028425 wait complete. 2023-12-06T03:52:49.582 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:49 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:48.310492+0000 mon.smithi132 (mon.0) 86 : audit [INF] from='client.? 172.21.15.132:0/4045072900' entity='client.admin' 2023-12-06T03:52:49.582 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:49 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:49.190082+0000 mon.smithi132 (mon.0) 87 : audit [INF] from='client.? 172.21.15.132:0/1174028425' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-12-06T03:52:49.703 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-12-06T03:52:50.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.073+0000 7ffaf917d700 1 Processor -- start 2023-12-06T03:52:50.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.073+0000 7ffaf917d700 1 -- start start 2023-12-06T03:52:50.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.073+0000 7ffaf917d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaf40703f0 0x7ffaf40fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:50.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.073+0000 7ffaf917d700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf40fee10 con 0x7ffaf40703f0 2023-12-06T03:52:50.076 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.074+0000 7ffaf2d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaf40703f0 0x7ffaf40fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:50.077 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.074+0000 7ffaf2d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaf40703f0 0x7ffaf40fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33284/0 (socket says 172.21.15.132:33284) 2023-12-06T03:52:50.077 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.074+0000 7ffaf2d9d700 1 -- 172.21.15.132:0/818890651 learned_addr learned my addr 172.21.15.132:0/818890651 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:50.077 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.074+0000 7ffaf2d9d700 1 -- 172.21.15.132:0/818890651 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffaf40fef50 con 0x7ffaf40703f0 2023-12-06T03:52:50.078 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.074+0000 7ffaf2d9d700 1 --2- 172.21.15.132:0/818890651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaf40703f0 0x7ffaf40fe920 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7ffae400b5f0 tx=0x7ffae4006ee0).ready entity=mon.0 client_cookie=cdf433e1ff270913 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:50.078 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.074+0000 7ffaf1d9b700 1 -- 172.21.15.132:0/818890651 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffae400d600 con 0x7ffaf40703f0 2023-12-06T03:52:50.078 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.074+0000 7ffaf1d9b700 1 -- 172.21.15.132:0/818890651 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffae400dc10 con 0x7ffaf40703f0 2023-12-06T03:52:50.078 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.074+0000 7ffaf1d9b700 1 -- 172.21.15.132:0/818890651 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffae4011cd0 con 0x7ffaf40703f0 2023-12-06T03:52:50.078 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.075+0000 7ffaf917d700 1 -- 172.21.15.132:0/818890651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaf40703f0 msgr2=0x7ffaf40fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:50.078 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.075+0000 7ffaf917d700 1 --2- 172.21.15.132:0/818890651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaf40703f0 0x7ffaf40fe920 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7ffae400b5f0 tx=0x7ffae4006ee0).stop 2023-12-06T03:52:50.078 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.075+0000 7ffaf917d700 1 -- 172.21.15.132:0/818890651 shutdown_connections 2023-12-06T03:52:50.078 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.075+0000 7ffaf917d700 1 --2- 172.21.15.132:0/818890651 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaf40703f0 0x7ffaf40fe920 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:50.080 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.075+0000 7ffaf917d700 1 -- 172.21.15.132:0/818890651 >> 172.21.15.132:0/818890651 conn(0x7ffaf40f63c0 msgr2=0x7ffaf40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:50.080 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.075+0000 7ffaf917d700 1 -- 172.21.15.132:0/818890651 shutdown_connections 2023-12-06T03:52:50.080 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.075+0000 7ffaf917d700 1 -- 172.21.15.132:0/818890651 wait complete. 2023-12-06T03:52:50.080 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.076+0000 7ffaf917d700 1 Processor -- start 2023-12-06T03:52:50.080 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.076+0000 7ffaf917d700 1 -- start start 2023-12-06T03:52:50.081 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.076+0000 7ffaf917d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaf40703f0 0x7ffaf410cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:50.081 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.076+0000 7ffaf917d700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffaf410d210 con 0x7ffaf40703f0 2023-12-06T03:52:50.081 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.076+0000 7ffaf2d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaf40703f0 0x7ffaf410cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:50.081 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.076+0000 7ffaf2d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaf40703f0 0x7ffaf410cd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33300/0 (socket says 172.21.15.132:33300) 2023-12-06T03:52:50.081 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.076+0000 7ffaf2d9d700 1 -- 172.21.15.132:0/3324842503 learned_addr learned my addr 172.21.15.132:0/3324842503 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:50.081 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.076+0000 7ffaf2d9d700 1 -- 172.21.15.132:0/3324842503 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae400b200 con 0x7ffaf40703f0 2023-12-06T03:52:50.081 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.076+0000 7ffaf2d9d700 1 --2- 172.21.15.132:0/3324842503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaf40703f0 0x7ffaf410cd20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7ffae4006a40 tx=0x7ffae4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:50.081 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.076+0000 7ffadb7fe700 1 -- 172.21.15.132:0/3324842503 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffae4011c90 con 0x7ffaf40703f0 2023-12-06T03:52:50.081 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.076+0000 7ffadb7fe700 1 -- 172.21.15.132:0/3324842503 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffae401b8c0 con 0x7ffaf40703f0 2023-12-06T03:52:50.081 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.076+0000 7ffaf917d700 1 -- 172.21.15.132:0/3324842503 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffaf410d410 con 0x7ffaf40703f0 2023-12-06T03:52:50.081 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.076+0000 7ffadb7fe700 1 -- 172.21.15.132:0/3324842503 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffae401a8c0 con 0x7ffaf40703f0 2023-12-06T03:52:50.081 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.077+0000 7ffaf917d700 1 -- 172.21.15.132:0/3324842503 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffaf410d820 con 0x7ffaf40703f0 2023-12-06T03:52:50.081 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.077+0000 7ffadb7fe700 1 -- 172.21.15.132:0/3324842503 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7ffae400f070 con 0x7ffaf40703f0 2023-12-06T03:52:50.081 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.077+0000 7ffadb7fe700 1 --2- 172.21.15.132:0/3324842503 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7ffae002feb0 0x7ffae0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:50.082 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.077+0000 7ffadb7fe700 1 -- 172.21.15.132:0/3324842503 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7ffae4041a00 con 0x7ffaf40703f0 2023-12-06T03:52:50.082 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.077+0000 7ffaf259c700 1 -- 172.21.15.132:0/3324842503 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7ffae002feb0 msgr2=0x7ffae0032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/597241079 2023-12-06T03:52:50.082 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.078+0000 7ffaf259c700 1 --2- 172.21.15.132:0/3324842503 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7ffae002feb0 0x7ffae0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-06T03:52:50.082 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.078+0000 7ffaf917d700 1 -- 172.21.15.132:0/3324842503 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffaf40ffc40 con 0x7ffaf40703f0 2023-12-06T03:52:50.084 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.082+0000 7ffadb7fe700 1 -- 172.21.15.132:0/3324842503 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffae4016070 con 0x7ffaf40703f0 2023-12-06T03:52:50.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.274+0000 7ffaf917d700 1 -- 172.21.15.132:0/3324842503 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7ffaf410e650 con 0x7ffaf40703f0 2023-12-06T03:52:50.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.275+0000 7ffadb7fe700 1 -- 172.21.15.132:0/3324842503 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v12) v1 ==== 56+0+104 (secure 0 0 0) 0x7ffae4010950 con 0x7ffaf40703f0 2023-12-06T03:52:50.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: { 2023-12-06T03:52:50.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "epoch": 12, 2023-12-06T03:52:50.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "available": true, 2023-12-06T03:52:50.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "active_name": "smithi132.gokxja", 2023-12-06T03:52:50.278 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-06T03:52:50.278 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: } 2023-12-06T03:52:50.279 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.277+0000 7ffaf917d700 1 -- 172.21.15.132:0/3324842503 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7ffae002feb0 msgr2=0x7ffae0032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-06T03:52:50.280 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.277+0000 7ffaf917d700 1 --2- 172.21.15.132:0/3324842503 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7ffae002feb0 0x7ffae0032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-06T03:52:50.280 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.277+0000 7ffaf917d700 1 -- 172.21.15.132:0/3324842503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaf40703f0 msgr2=0x7ffaf410cd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:50.280 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.277+0000 7ffaf917d700 1 --2- 172.21.15.132:0/3324842503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaf40703f0 0x7ffaf410cd20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7ffae4006a40 tx=0x7ffae4006ee0).stop 2023-12-06T03:52:50.280 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.277+0000 7ffaf917d700 1 -- 172.21.15.132:0/3324842503 shutdown_connections 2023-12-06T03:52:50.280 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.277+0000 7ffaf917d700 1 --2- 172.21.15.132:0/3324842503 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7ffae002feb0 0x7ffae0032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-06T03:52:50.280 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.277+0000 7ffaf917d700 1 --2- 172.21.15.132:0/3324842503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffaf40703f0 0x7ffaf410cd20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:50.280 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.277+0000 7ffaf917d700 1 -- 172.21.15.132:0/3324842503 >> 172.21.15.132:0/3324842503 conn(0x7ffaf40f63c0 msgr2=0x7ffaf40f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:50.280 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.277+0000 7ffaf917d700 1 -- 172.21.15.132:0/3324842503 shutdown_connections 2023-12-06T03:52:50.280 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:50.277+0000 7ffaf917d700 1 -- 172.21.15.132:0/3324842503 wait complete. 2023-12-06T03:52:50.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:50 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:49.316863+0000 mon.smithi132 (mon.0) 88 : audit [INF] from='client.? 172.21.15.132:0/1174028425' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-12-06T03:52:50.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:50 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:49.316938+0000 mon.smithi132 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi132.gokxja(active, since 6s) 2023-12-06T03:52:50.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:50 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:50.276420+0000 mon.smithi132 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.132:0/3324842503' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-06T03:52:50.810 INFO:teuthology.orchestra.run.smithi132.stderr:Waiting for the mgr to restart... 2023-12-06T03:52:50.810 INFO:teuthology.orchestra.run.smithi132.stderr:Waiting for mgr epoch 12... 2023-12-06T03:52:50.810 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-12-06T03:52:51.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.176+0000 7fa6eb9d4700 1 Processor -- start 2023-12-06T03:52:51.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.177+0000 7fa6eb9d4700 1 -- start start 2023-12-06T03:52:51.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.177+0000 7fa6eb9d4700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6e40fd350 0x7fa6e40fd770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:51.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.177+0000 7fa6eb9d4700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e40fdc60 con 0x7fa6e40fd350 2023-12-06T03:52:51.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.177+0000 7fa6e9770700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6e40fd350 0x7fa6e40fd770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:51.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.177+0000 7fa6e9770700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6e40fd350 0x7fa6e40fd770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33308/0 (socket says 172.21.15.132:33308) 2023-12-06T03:52:51.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.177+0000 7fa6e9770700 1 -- 172.21.15.132:0/679575508 learned_addr learned my addr 172.21.15.132:0/679575508 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:51.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.177+0000 7fa6e9770700 1 -- 172.21.15.132:0/679575508 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e40fdda0 con 0x7fa6e40fd350 2023-12-06T03:52:51.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.177+0000 7fa6e9770700 1 --2- 172.21.15.132:0/679575508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6e40fd350 0x7fa6e40fd770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa6e000fa00 tx=0x7fa6e0006ee0).ready entity=mon.0 client_cookie=9ac15e732f3cd22c server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:51.182 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.177+0000 7fa6d3fff700 1 -- 172.21.15.132:0/679575508 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e000f3a0 con 0x7fa6e40fd350 2023-12-06T03:52:51.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.178+0000 7fa6d3fff700 1 -- 172.21.15.132:0/679575508 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6e0009ae0 con 0x7fa6e40fd350 2023-12-06T03:52:51.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.178+0000 7fa6d3fff700 1 -- 172.21.15.132:0/679575508 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e000ebc0 con 0x7fa6e40fd350 2023-12-06T03:52:51.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.178+0000 7fa6eb9d4700 1 -- 172.21.15.132:0/679575508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6e40fd350 msgr2=0x7fa6e40fd770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:51.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.178+0000 7fa6eb9d4700 1 --2- 172.21.15.132:0/679575508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6e40fd350 0x7fa6e40fd770 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa6e000fa00 tx=0x7fa6e0006ee0).stop 2023-12-06T03:52:51.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.178+0000 7fa6eb9d4700 1 -- 172.21.15.132:0/679575508 shutdown_connections 2023-12-06T03:52:51.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.178+0000 7fa6eb9d4700 1 --2- 172.21.15.132:0/679575508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6e40fd350 0x7fa6e40fd770 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:51.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.178+0000 7fa6eb9d4700 1 -- 172.21.15.132:0/679575508 >> 172.21.15.132:0/679575508 conn(0x7fa6e40f5b10 msgr2=0x7fa6e40f7f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:51.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.178+0000 7fa6eb9d4700 1 -- 172.21.15.132:0/679575508 shutdown_connections 2023-12-06T03:52:51.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.178+0000 7fa6eb9d4700 1 -- 172.21.15.132:0/679575508 wait complete. 2023-12-06T03:52:51.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.179+0000 7fa6eb9d4700 1 Processor -- start 2023-12-06T03:52:51.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.179+0000 7fa6eb9d4700 1 -- start start 2023-12-06T03:52:51.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.179+0000 7fa6eb9d4700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6e40fd350 0x7fa6e41087c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:51.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.179+0000 7fa6eb9d4700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6e4108cb0 con 0x7fa6e40fd350 2023-12-06T03:52:51.183 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.179+0000 7fa6e9770700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6e40fd350 0x7fa6e41087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:51.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.179+0000 7fa6e9770700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6e40fd350 0x7fa6e41087c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33316/0 (socket says 172.21.15.132:33316) 2023-12-06T03:52:51.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.179+0000 7fa6e9770700 1 -- 172.21.15.132:0/214144656 learned_addr learned my addr 172.21.15.132:0/214144656 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:51.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.180+0000 7fa6e9770700 1 -- 172.21.15.132:0/214144656 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6e000f610 con 0x7fa6e40fd350 2023-12-06T03:52:51.184 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.180+0000 7fa6e9770700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6e40fd350 0x7fa6e41087c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fa6e0009e80 tx=0x7fa6e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:51.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.180+0000 7fa6d1ffb700 1 -- 172.21.15.132:0/214144656 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e0021440 con 0x7fa6e40fd350 2023-12-06T03:52:51.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.180+0000 7fa6d1ffb700 1 -- 172.21.15.132:0/214144656 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa6e0021a50 con 0x7fa6e40fd350 2023-12-06T03:52:51.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.180+0000 7fa6eb9d4700 1 -- 172.21.15.132:0/214144656 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6e4108eb0 con 0x7fa6e40fd350 2023-12-06T03:52:51.185 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.180+0000 7fa6d1ffb700 1 -- 172.21.15.132:0/214144656 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa6e0018900 con 0x7fa6e40fd350 2023-12-06T03:52:51.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.180+0000 7fa6eb9d4700 1 -- 172.21.15.132:0/214144656 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6e41092c0 con 0x7fa6e40fd350 2023-12-06T03:52:51.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.181+0000 7fa6d1ffb700 1 -- 172.21.15.132:0/214144656 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 12) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fa6e0018a60 con 0x7fa6e40fd350 2023-12-06T03:52:51.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.181+0000 7fa6d1ffb700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa6cc02ff00 0x7fa6cc0323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:51.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.181+0000 7fa6d1ffb700 1 -- 172.21.15.132:0/214144656 --> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6cc032a80 con 0x7fa6cc02ff00 2023-12-06T03:52:51.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.181+0000 7fa6d1ffb700 1 -- 172.21.15.132:0/214144656 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fa6e0045170 con 0x7fa6e40fd350 2023-12-06T03:52:51.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.181+0000 7fa6e8f6f700 1 -- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa6cc02ff00 msgr2=0x7fa6cc0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/597241079 2023-12-06T03:52:51.186 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.181+0000 7fa6e8f6f700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa6cc02ff00 0x7fa6cc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-06T03:52:51.384 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.382+0000 7fa6e8f6f700 1 -- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa6cc02ff00 msgr2=0x7fa6cc0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/597241079 2023-12-06T03:52:51.384 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.382+0000 7fa6e8f6f700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa6cc02ff00 0x7fa6cc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-06T03:52:51.785 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.783+0000 7fa6e8f6f700 1 -- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa6cc02ff00 msgr2=0x7fa6cc0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/597241079 2023-12-06T03:52:51.785 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:51.783+0000 7fa6e8f6f700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa6cc02ff00 0x7fa6cc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-06T03:52:52.586 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:52.584+0000 7fa6e8f6f700 1 -- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa6cc02ff00 msgr2=0x7fa6cc0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/597241079 2023-12-06T03:52:52.586 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:52.584+0000 7fa6e8f6f700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa6cc02ff00 0x7fa6cc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-06T03:52:54.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:54.185+0000 7fa6e8f6f700 1 -- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa6cc02ff00 msgr2=0x7fa6cc0323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.132:6800/597241079 2023-12-06T03:52:54.188 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:54.185+0000 7fa6e8f6f700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa6cc02ff00 0x7fa6cc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-06T03:52:54.377 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:54.375+0000 7fa6d1ffb700 1 -- 172.21.15.132:0/214144656 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mgrmap(e 13) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fa6e002a3f0 con 0x7fa6e40fd350 2023-12-06T03:52:54.378 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:54.375+0000 7fa6d1ffb700 1 -- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa6cc02ff00 msgr2=0x7fa6cc0323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-06T03:52:54.378 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:54.375+0000 7fa6d1ffb700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa6cc02ff00 0x7fa6cc0323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-06T03:52:54.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:54.324539+0000 mon.smithi132 (mon.0) 91 : cluster [INF] Active manager daemon smithi132.gokxja restarted 2023-12-06T03:52:54.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:54.324916+0000 mon.smithi132 (mon.0) 92 : cluster [INF] Activating manager daemon smithi132.gokxja 2023-12-06T03:52:54.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:54.326567+0000 mon.smithi132 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-06T03:52:55.384 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.382+0000 7fa6d1ffb700 1 -- 172.21.15.132:0/214144656 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fa6e00203a0 con 0x7fa6e40fd350 2023-12-06T03:52:55.384 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.382+0000 7fa6d1ffb700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa6cc033b90 0x7fa6cc035f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:55.385 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.382+0000 7fa6d1ffb700 1 -- 172.21.15.132:0/214144656 --> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa6cc032a80 con 0x7fa6cc033b90 2023-12-06T03:52:55.385 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.383+0000 7fa6e8f6f700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa6cc033b90 0x7fa6cc035f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:55.385 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.383+0000 7fa6e8f6f700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa6cc033b90 0x7fa6cc035f80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fa6d4006fd0 tx=0x7fa6d4006d70).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:55.385 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.384+0000 7fa6d1ffb700 1 -- 172.21.15.132:0/214144656 <== mgr.14162 v2:172.21.15.132:6800/1042262986 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fa6cc032a80 con 0x7fa6cc033b90 2023-12-06T03:52:55.393 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.390+0000 7fa6eb9d4700 1 -- 172.21.15.132:0/214144656 --> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fa6e4059100 con 0x7fa6cc033b90 2023-12-06T03:52:55.394 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.390+0000 7fa6d1ffb700 1 -- 172.21.15.132:0/214144656 <== mgr.14162 v2:172.21.15.132:6800/1042262986 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fa6e4059100 con 0x7fa6cc033b90 2023-12-06T03:52:55.394 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: { 2023-12-06T03:52:55.394 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "mgrmap_epoch": 14, 2023-12-06T03:52:55.394 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: "initialized": true 2023-12-06T03:52:55.394 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: } 2023-12-06T03:52:55.394 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.391+0000 7fa6eb9d4700 1 -- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa6cc033b90 msgr2=0x7fa6cc035f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:55.394 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.391+0000 7fa6eb9d4700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa6cc033b90 0x7fa6cc035f80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fa6d4006fd0 tx=0x7fa6d4006d70).stop 2023-12-06T03:52:55.394 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.391+0000 7fa6eb9d4700 1 -- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6e40fd350 msgr2=0x7fa6e41087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:55.394 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.391+0000 7fa6eb9d4700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6e40fd350 0x7fa6e41087c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fa6e0009e80 tx=0x7fa6e0006ee0).stop 2023-12-06T03:52:55.394 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.391+0000 7fa6eb9d4700 1 -- 172.21.15.132:0/214144656 shutdown_connections 2023-12-06T03:52:55.394 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.391+0000 7fa6eb9d4700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa6cc033b90 0x7fa6cc035f80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:55.394 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.391+0000 7fa6eb9d4700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6e40fd350 0x7fa6e41087c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:55.394 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.391+0000 7fa6eb9d4700 1 --2- 172.21.15.132:0/214144656 >> [v2:172.21.15.132:6800/597241079,v1:172.21.15.132:6801/597241079] conn(0x7fa6cc02ff00 0x7fa6cc0323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-06T03:52:55.395 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.391+0000 7fa6eb9d4700 1 -- 172.21.15.132:0/214144656 >> 172.21.15.132:0/214144656 conn(0x7fa6e40f5b10 msgr2=0x7fa6e40f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:55.395 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.391+0000 7fa6eb9d4700 1 -- 172.21.15.132:0/214144656 shutdown_connections 2023-12-06T03:52:55.395 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:55.391+0000 7fa6eb9d4700 1 -- 172.21.15.132:0/214144656 wait complete. 2023-12-06T03:52:55.558 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:54.377865+0000 mon.smithi132 (mon.0) 94 : cluster [DBG] mgrmap e13: smithi132.gokxja(active, starting, since 0.0530349s) 2023-12-06T03:52:55.558 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:54.379269+0000 mon.smithi132 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mon metadata", "id": "smithi132"}]: dispatch 2023-12-06T03:52:55.559 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:54.379894+0000 mon.smithi132 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mgr metadata", "who": "smithi132.gokxja", "id": "smithi132.gokxja"}]: dispatch 2023-12-06T03:52:55.559 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:54.382511+0000 mon.smithi132 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-06T03:52:55.559 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:54.382700+0000 mon.smithi132 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-06T03:52:55.559 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:54.382832+0000 mon.smithi132 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-06T03:52:55.559 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:54.396945+0000 mon.smithi132 (mon.0) 100 : cluster [INF] Manager daemon smithi132.gokxja is now available 2023-12-06T03:52:55.559 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:54.496078+0000 mon.smithi132 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-06T03:52:55.559 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:54.497521+0000 mon.smithi132 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-06T03:52:55.559 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:54.514475+0000 mon.smithi132 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/mirror_snapshot_schedule"}]: dispatch 2023-12-06T03:52:55.559 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:54.523428+0000 mon.smithi132 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/trash_purge_schedule"}]: dispatch 2023-12-06T03:52:55.559 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:55.364171+0000 mon.smithi132 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:55.818 INFO:teuthology.orchestra.run.smithi132.stderr:mgr epoch 12 is available 2023-12-06T03:52:55.818 INFO:teuthology.orchestra.run.smithi132.stderr:Generating a dashboard self-signed certificate... 2023-12-06T03:52:55.818 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-12-06T03:52:56.274 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.267+0000 7f17f2e16700 1 Processor -- start 2023-12-06T03:52:56.275 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.267+0000 7f17f2e16700 1 -- start start 2023-12-06T03:52:56.275 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.268+0000 7f17f2e16700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0fc290 0x7f17ec0fc6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:56.275 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.268+0000 7f17f2e16700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec0fcba0 con 0x7f17ec0fc290 2023-12-06T03:52:56.275 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.268+0000 7f17f0bb2700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0fc290 0x7f17ec0fc6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:56.275 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.268+0000 7f17f0bb2700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0fc290 0x7f17ec0fc6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33368/0 (socket says 172.21.15.132:33368) 2023-12-06T03:52:56.275 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.268+0000 7f17f0bb2700 1 -- 172.21.15.132:0/1530688679 learned_addr learned my addr 172.21.15.132:0/1530688679 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:56.275 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.268+0000 7f17f0bb2700 1 -- 172.21.15.132:0/1530688679 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ec0fcce0 con 0x7f17ec0fc290 2023-12-06T03:52:56.275 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.268+0000 7f17f0bb2700 1 --2- 172.21.15.132:0/1530688679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0fc290 0x7f17ec0fc6b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f17e000b5f0 tx=0x7f17e0006ee0).ready entity=mon.0 client_cookie=d5050ad82805156e server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:56.275 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.268+0000 7f17eb7fe700 1 -- 172.21.15.132:0/1530688679 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e000d600 con 0x7f17ec0fc290 2023-12-06T03:52:56.275 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.268+0000 7f17eb7fe700 1 -- 172.21.15.132:0/1530688679 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17e000dc10 con 0x7f17ec0fc290 2023-12-06T03:52:56.275 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.268+0000 7f17eb7fe700 1 -- 172.21.15.132:0/1530688679 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e0011cd0 con 0x7f17ec0fc290 2023-12-06T03:52:56.275 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.269+0000 7f17f2e16700 1 -- 172.21.15.132:0/1530688679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0fc290 msgr2=0x7f17ec0fc6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:56.275 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.269+0000 7f17f2e16700 1 --2- 172.21.15.132:0/1530688679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0fc290 0x7f17ec0fc6b0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f17e000b5f0 tx=0x7f17e0006ee0).stop 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.269+0000 7f17f2e16700 1 -- 172.21.15.132:0/1530688679 shutdown_connections 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.269+0000 7f17f2e16700 1 --2- 172.21.15.132:0/1530688679 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0fc290 0x7f17ec0fc6b0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.269+0000 7f17f2e16700 1 -- 172.21.15.132:0/1530688679 >> 172.21.15.132:0/1530688679 conn(0x7f17ec0f5b40 msgr2=0x7f17ec0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.269+0000 7f17f2e16700 1 -- 172.21.15.132:0/1530688679 shutdown_connections 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.269+0000 7f17f2e16700 1 -- 172.21.15.132:0/1530688679 wait complete. 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.270+0000 7f17f2e16700 1 Processor -- start 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.270+0000 7f17f2e16700 1 -- start start 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.270+0000 7f17f2e16700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0fc290 0x7f17ec10cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.270+0000 7f17f2e16700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec10d100 con 0x7f17ec0fc290 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.270+0000 7f17f0bb2700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0fc290 0x7f17ec10cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.270+0000 7f17f0bb2700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0fc290 0x7f17ec10cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33374/0 (socket says 172.21.15.132:33374) 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.270+0000 7f17f0bb2700 1 -- 172.21.15.132:0/304953339 learned_addr learned my addr 172.21.15.132:0/304953339 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.271+0000 7f17f0bb2700 1 -- 172.21.15.132:0/304953339 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17e000b200 con 0x7f17ec0fc290 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.271+0000 7f17f0bb2700 1 --2- 172.21.15.132:0/304953339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0fc290 0x7f17ec10cc10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f17e000b1d0 tx=0x7f17e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:56.276 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.271+0000 7f17e97fa700 1 -- 172.21.15.132:0/304953339 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e0011df0 con 0x7f17ec0fc290 2023-12-06T03:52:56.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.271+0000 7f17f2e16700 1 -- 172.21.15.132:0/304953339 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ec10d300 con 0x7f17ec0fc290 2023-12-06T03:52:56.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.271+0000 7f17e97fa700 1 -- 172.21.15.132:0/304953339 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17e001b8c0 con 0x7f17ec0fc290 2023-12-06T03:52:56.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.271+0000 7f17e97fa700 1 -- 172.21.15.132:0/304953339 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17e001a870 con 0x7f17ec0fc290 2023-12-06T03:52:56.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.271+0000 7f17f2e16700 1 -- 172.21.15.132:0/304953339 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ec10d710 con 0x7f17ec0fc290 2023-12-06T03:52:56.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.272+0000 7f17e97fa700 1 -- 172.21.15.132:0/304953339 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 14) v1 ==== 36067+0+0 (secure 0 0 0) 0x7f17e000f070 con 0x7f17ec0fc290 2023-12-06T03:52:56.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.272+0000 7f17e97fa700 1 --2- 172.21.15.132:0/304953339 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f17d002fd70 0x7f17d0032230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:56.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.272+0000 7f17e97fa700 1 -- 172.21.15.132:0/304953339 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17e001b560 con 0x7f17ec0fc290 2023-12-06T03:52:56.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.272+0000 7f17f2e16700 1 -- 172.21.15.132:0/304953339 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ec045ad0 con 0x7f17ec0fc290 2023-12-06T03:52:56.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.272+0000 7f17ebfff700 1 --2- 172.21.15.132:0/304953339 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f17d002fd70 0x7f17d0032230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:56.277 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.272+0000 7f17ebfff700 1 --2- 172.21.15.132:0/304953339 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f17d002fd70 0x7f17d0032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f17d80096a0 tx=0x7f17d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:56.278 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.276+0000 7f17e97fa700 1 -- 172.21.15.132:0/304953339 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17e0016070 con 0x7f17ec0fc290 2023-12-06T03:52:56.391 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.389+0000 7f17e97fa700 1 -- 172.21.15.132:0/304953339 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f17e0024790 con 0x7f17ec0fc290 2023-12-06T03:52:56.439 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.437+0000 7f17f2e16700 1 -- 172.21.15.132:0/304953339 --> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f17ec106ba0 con 0x7f17d002fd70 2023-12-06T03:52:56.512 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.509+0000 7f17e97fa700 1 -- 172.21.15.132:0/304953339 <== mgr.14162 v2:172.21.15.132:6800/1042262986 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f17ec106ba0 con 0x7f17d002fd70 2023-12-06T03:52:56.512 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: Self-signed certificate created 2023-12-06T03:52:56.514 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.511+0000 7f17f2e16700 1 -- 172.21.15.132:0/304953339 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f17d002fd70 msgr2=0x7f17d0032230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:56.514 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.511+0000 7f17f2e16700 1 --2- 172.21.15.132:0/304953339 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f17d002fd70 0x7f17d0032230 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f17d80096a0 tx=0x7f17d8006b40).stop 2023-12-06T03:52:56.514 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.511+0000 7f17f2e16700 1 -- 172.21.15.132:0/304953339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0fc290 msgr2=0x7f17ec10cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:56.514 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.511+0000 7f17f2e16700 1 --2- 172.21.15.132:0/304953339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0fc290 0x7f17ec10cc10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f17e000b1d0 tx=0x7f17e0006ee0).stop 2023-12-06T03:52:56.514 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.511+0000 7f17f2e16700 1 -- 172.21.15.132:0/304953339 shutdown_connections 2023-12-06T03:52:56.514 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.511+0000 7f17f2e16700 1 --2- 172.21.15.132:0/304953339 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f17d002fd70 0x7f17d0032230 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:56.514 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.511+0000 7f17f2e16700 1 --2- 172.21.15.132:0/304953339 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0fc290 0x7f17ec10cc10 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:56.514 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.511+0000 7f17f2e16700 1 -- 172.21.15.132:0/304953339 >> 172.21.15.132:0/304953339 conn(0x7f17ec0f5b40 msgr2=0x7f17ec0f75e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:56.514 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.511+0000 7f17f2e16700 1 -- 172.21.15.132:0/304953339 shutdown_connections 2023-12-06T03:52:56.515 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:56.511+0000 7f17f2e16700 1 -- 172.21.15.132:0/304953339 wait complete. 2023-12-06T03:52:56.666 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:56 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:55.385267+0000 mgr.smithi132.gokxja (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-06T03:52:56.667 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:56 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:55.385382+0000 mon.smithi132 (mon.0) 106 : cluster [DBG] mgrmap e14: smithi132.gokxja(active, since 1.06055s) 2023-12-06T03:52:56.667 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:56 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:55.392257+0000 mgr.smithi132.gokxja (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-06T03:52:56.819 INFO:teuthology.orchestra.run.smithi132.stderr:Creating initial admin user... 2023-12-06T03:52:56.819 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpaodj8frd:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-12-06T03:52:57.144 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.138+0000 7f3e701e8700 1 Processor -- start 2023-12-06T03:52:57.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.138+0000 7f3e701e8700 1 -- start start 2023-12-06T03:52:57.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.138+0000 7f3e701e8700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e680703b0 0x7f3e68100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:57.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.138+0000 7f3e701e8700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e680fc750 con 0x7f3e680703b0 2023-12-06T03:52:57.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.138+0000 7f3e6df84700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e680703b0 0x7f3e68100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:57.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.138+0000 7f3e6df84700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e680703b0 0x7f3e68100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33390/0 (socket says 172.21.15.132:33390) 2023-12-06T03:52:57.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.138+0000 7f3e6df84700 1 -- 172.21.15.132:0/3220152290 learned_addr learned my addr 172.21.15.132:0/3220152290 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:57.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.138+0000 7f3e6df84700 1 -- 172.21.15.132:0/3220152290 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e68101250 con 0x7f3e680703b0 2023-12-06T03:52:57.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.139+0000 7f3e6df84700 1 --2- 172.21.15.132:0/3220152290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e680703b0 0x7f3e68100d60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3e6400b5f0 tx=0x7f3e64006ee0).ready entity=mon.0 client_cookie=f011d988877d8358 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:57.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.139+0000 7f3e6cf82700 1 -- 172.21.15.132:0/3220152290 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e6400d600 con 0x7f3e680703b0 2023-12-06T03:52:57.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.139+0000 7f3e6cf82700 1 -- 172.21.15.132:0/3220152290 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e6400dc10 con 0x7f3e680703b0 2023-12-06T03:52:57.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.139+0000 7f3e6cf82700 1 -- 172.21.15.132:0/3220152290 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e64011cd0 con 0x7f3e680703b0 2023-12-06T03:52:57.145 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.139+0000 7f3e701e8700 1 -- 172.21.15.132:0/3220152290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e680703b0 msgr2=0x7f3e68100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:57.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.139+0000 7f3e701e8700 1 --2- 172.21.15.132:0/3220152290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e680703b0 0x7f3e68100d60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3e6400b5f0 tx=0x7f3e64006ee0).stop 2023-12-06T03:52:57.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.139+0000 7f3e701e8700 1 -- 172.21.15.132:0/3220152290 shutdown_connections 2023-12-06T03:52:57.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.139+0000 7f3e701e8700 1 --2- 172.21.15.132:0/3220152290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e680703b0 0x7f3e68100d60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:57.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.139+0000 7f3e701e8700 1 -- 172.21.15.132:0/3220152290 >> 172.21.15.132:0/3220152290 conn(0x7f3e680f63c0 msgr2=0x7f3e680f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:57.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.139+0000 7f3e701e8700 1 -- 172.21.15.132:0/3220152290 shutdown_connections 2023-12-06T03:52:57.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.139+0000 7f3e701e8700 1 -- 172.21.15.132:0/3220152290 wait complete. 2023-12-06T03:52:57.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.140+0000 7f3e701e8700 1 Processor -- start 2023-12-06T03:52:57.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.140+0000 7f3e701e8700 1 -- start start 2023-12-06T03:52:57.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.140+0000 7f3e701e8700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e680703b0 0x7f3e6810cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:57.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.140+0000 7f3e701e8700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e6810d200 con 0x7f3e680703b0 2023-12-06T03:52:57.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.140+0000 7f3e6df84700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e680703b0 0x7f3e6810cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:57.146 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.140+0000 7f3e6df84700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e680703b0 0x7f3e6810cd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:33396/0 (socket says 172.21.15.132:33396) 2023-12-06T03:52:57.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.140+0000 7f3e6df84700 1 -- 172.21.15.132:0/2129162781 learned_addr learned my addr 172.21.15.132:0/2129162781 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:57.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.140+0000 7f3e6df84700 1 -- 172.21.15.132:0/2129162781 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e6400b200 con 0x7f3e680703b0 2023-12-06T03:52:57.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.140+0000 7f3e6df84700 1 --2- 172.21.15.132:0/2129162781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e680703b0 0x7f3e6810cd10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3e6400dfb0 tx=0x7f3e64006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:57.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.141+0000 7f3e567fc700 1 -- 172.21.15.132:0/2129162781 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e6401b410 con 0x7f3e680703b0 2023-12-06T03:52:57.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.141+0000 7f3e567fc700 1 -- 172.21.15.132:0/2129162781 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3e6401ba20 con 0x7f3e680703b0 2023-12-06T03:52:57.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.141+0000 7f3e701e8700 1 -- 172.21.15.132:0/2129162781 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e6810d400 con 0x7f3e680703b0 2023-12-06T03:52:57.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.141+0000 7f3e567fc700 1 -- 172.21.15.132:0/2129162781 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e6401a970 con 0x7f3e680703b0 2023-12-06T03:52:57.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.141+0000 7f3e701e8700 1 -- 172.21.15.132:0/2129162781 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e6810d810 con 0x7f3e680703b0 2023-12-06T03:52:57.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.142+0000 7f3e567fc700 1 -- 172.21.15.132:0/2129162781 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f3e6400f070 con 0x7f3e680703b0 2023-12-06T03:52:57.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.142+0000 7f3e567fc700 1 --2- 172.21.15.132:0/2129162781 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3e5002feb0 0x7f3e50032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:57.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.142+0000 7f3e567fc700 1 -- 172.21.15.132:0/2129162781 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3e64041200 con 0x7f3e680703b0 2023-12-06T03:52:57.147 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.142+0000 7f3e701e8700 1 -- 172.21.15.132:0/2129162781 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e68045ad0 con 0x7f3e680703b0 2023-12-06T03:52:57.148 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.142+0000 7f3e6d783700 1 --2- 172.21.15.132:0/2129162781 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3e5002feb0 0x7f3e50032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:57.148 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.142+0000 7f3e6d783700 1 --2- 172.21.15.132:0/2129162781 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3e5002feb0 0x7f3e50032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3e580096a0 tx=0x7f3e58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:57.148 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.146+0000 7f3e567fc700 1 -- 172.21.15.132:0/2129162781 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3e64016070 con 0x7f3e680703b0 2023-12-06T03:52:57.315 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.312+0000 7f3e701e8700 1 -- 172.21.15.132:0/2129162781 --> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f3e68058f20 con 0x7f3e5002feb0 2023-12-06T03:52:57.532 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.530+0000 7f3e567fc700 1 -- 172.21.15.132:0/2129162781 <== mgr.14162 v2:172.21.15.132:6800/1042262986 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f3e68058f20 con 0x7f3e5002feb0 2023-12-06T03:52:57.532 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$sYfm7ENoiKQCVRAHKZKpP..LTQFr9vqWdWo2lWCTm7OMT10DfoSK.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1701834777, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-12-06T03:52:57.535 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.532+0000 7f3e701e8700 1 -- 172.21.15.132:0/2129162781 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3e5002feb0 msgr2=0x7f3e50032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:57.535 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.532+0000 7f3e701e8700 1 --2- 172.21.15.132:0/2129162781 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3e5002feb0 0x7f3e50032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f3e580096a0 tx=0x7f3e58006b40).stop 2023-12-06T03:52:57.535 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.532+0000 7f3e701e8700 1 -- 172.21.15.132:0/2129162781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e680703b0 msgr2=0x7f3e6810cd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:57.535 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.532+0000 7f3e701e8700 1 --2- 172.21.15.132:0/2129162781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e680703b0 0x7f3e6810cd10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3e6400dfb0 tx=0x7f3e64006ee0).stop 2023-12-06T03:52:57.535 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.532+0000 7f3e701e8700 1 -- 172.21.15.132:0/2129162781 shutdown_connections 2023-12-06T03:52:57.535 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.532+0000 7f3e701e8700 1 --2- 172.21.15.132:0/2129162781 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3e5002feb0 0x7f3e50032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:57.535 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.532+0000 7f3e701e8700 1 --2- 172.21.15.132:0/2129162781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3e680703b0 0x7f3e6810cd10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:57.535 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.532+0000 7f3e701e8700 1 -- 172.21.15.132:0/2129162781 >> 172.21.15.132:0/2129162781 conn(0x7f3e680f63c0 msgr2=0x7f3e680f7f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:57.535 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.532+0000 7f3e701e8700 1 -- 172.21.15.132:0/2129162781 shutdown_connections 2023-12-06T03:52:57.535 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:57.532+0000 7f3e701e8700 1 -- 172.21.15.132:0/2129162781 wait complete. 2023-12-06T03:52:57.724 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:52:56.391559+0000 mon.smithi132 (mon.0) 107 : cluster [DBG] mgrmap e15: smithi132.gokxja(active, since 2s) 2023-12-06T03:52:57.724 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:56.439721+0000 mgr.smithi132.gokxja (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-12-06T03:52:57.725 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:56.509260+0000 mon.smithi132 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:57.725 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:56.511001+0000 mon.smithi132 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:57.725 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:57.314311+0000 mgr.smithi132.gokxja (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-12-06T03:52:57.926 INFO:teuthology.orchestra.run.smithi132.stderr:Fetching dashboard port number... 2023-12-06T03:52:57.926 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpbucrmb9w:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpzstj0_lo:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-12-06T03:52:58.283 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.279+0000 7f24e236c700 1 Processor -- start 2023-12-06T03:52:58.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.279+0000 7f24e236c700 1 -- start start 2023-12-06T03:52:58.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.280+0000 7f24e236c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24dc0703b0 0x7f24dc100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:58.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.280+0000 7f24e236c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc0fc750 con 0x7f24dc0703b0 2023-12-06T03:52:58.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.280+0000 7f24dbfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24dc0703b0 0x7f24dc100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:58.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.280+0000 7f24dbfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24dc0703b0 0x7f24dc100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:55184/0 (socket says 172.21.15.132:55184) 2023-12-06T03:52:58.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.280+0000 7f24dbfff700 1 -- 172.21.15.132:0/1131677204 learned_addr learned my addr 172.21.15.132:0/1131677204 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:58.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.280+0000 7f24dbfff700 1 -- 172.21.15.132:0/1131677204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc101250 con 0x7f24dc0703b0 2023-12-06T03:52:58.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.280+0000 7f24dbfff700 1 --2- 172.21.15.132:0/1131677204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24dc0703b0 0x7f24dc100d60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f24cc00b5f0 tx=0x7f24cc006ee0).ready entity=mon.0 client_cookie=115b744550ef800c server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:58.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.280+0000 7f24daffd700 1 -- 172.21.15.132:0/1131677204 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24cc00d600 con 0x7f24dc0703b0 2023-12-06T03:52:58.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.280+0000 7f24daffd700 1 -- 172.21.15.132:0/1131677204 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24cc00dc10 con 0x7f24dc0703b0 2023-12-06T03:52:58.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.280+0000 7f24daffd700 1 -- 172.21.15.132:0/1131677204 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24cc011cd0 con 0x7f24dc0703b0 2023-12-06T03:52:58.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.281+0000 7f24e236c700 1 -- 172.21.15.132:0/1131677204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24dc0703b0 msgr2=0x7f24dc100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:58.284 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.281+0000 7f24e236c700 1 --2- 172.21.15.132:0/1131677204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24dc0703b0 0x7f24dc100d60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f24cc00b5f0 tx=0x7f24cc006ee0).stop 2023-12-06T03:52:58.285 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.281+0000 7f24e236c700 1 -- 172.21.15.132:0/1131677204 shutdown_connections 2023-12-06T03:52:58.285 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.281+0000 7f24e236c700 1 --2- 172.21.15.132:0/1131677204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24dc0703b0 0x7f24dc100d60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:58.285 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.281+0000 7f24e236c700 1 -- 172.21.15.132:0/1131677204 >> 172.21.15.132:0/1131677204 conn(0x7f24dc0f63c0 msgr2=0x7f24dc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:58.285 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.281+0000 7f24e236c700 1 -- 172.21.15.132:0/1131677204 shutdown_connections 2023-12-06T03:52:58.285 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.281+0000 7f24e236c700 1 -- 172.21.15.132:0/1131677204 wait complete. 2023-12-06T03:52:58.287 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.282+0000 7f24e236c700 1 Processor -- start 2023-12-06T03:52:58.287 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.282+0000 7f24e236c700 1 -- start start 2023-12-06T03:52:58.287 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.282+0000 7f24e236c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24dc0703b0 0x7f24dc10cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:58.287 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.282+0000 7f24e236c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc10d230 con 0x7f24dc0703b0 2023-12-06T03:52:58.287 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.282+0000 7f24dbfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24dc0703b0 0x7f24dc10cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:58.287 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.282+0000 7f24dbfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24dc0703b0 0x7f24dc10cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:55196/0 (socket says 172.21.15.132:55196) 2023-12-06T03:52:58.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.282+0000 7f24dbfff700 1 -- 172.21.15.132:0/1751406949 learned_addr learned my addr 172.21.15.132:0/1751406949 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:52:58.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.283+0000 7f24dbfff700 1 -- 172.21.15.132:0/1751406949 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24cc00b200 con 0x7f24dc0703b0 2023-12-06T03:52:58.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.283+0000 7f24dbfff700 1 --2- 172.21.15.132:0/1751406949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24dc0703b0 0x7f24dc10cd40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f24cc00dfb0 tx=0x7f24cc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:58.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.283+0000 7f24d8ff9700 1 -- 172.21.15.132:0/1751406949 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24cc01b410 con 0x7f24dc0703b0 2023-12-06T03:52:58.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.283+0000 7f24d8ff9700 1 -- 172.21.15.132:0/1751406949 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24cc01ba20 con 0x7f24dc0703b0 2023-12-06T03:52:58.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.283+0000 7f24d8ff9700 1 -- 172.21.15.132:0/1751406949 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24cc01a970 con 0x7f24dc0703b0 2023-12-06T03:52:58.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.283+0000 7f24e236c700 1 -- 172.21.15.132:0/1751406949 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24dc10d430 con 0x7f24dc0703b0 2023-12-06T03:52:58.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.283+0000 7f24e236c700 1 -- 172.21.15.132:0/1751406949 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24dc10d840 con 0x7f24dc0703b0 2023-12-06T03:52:58.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.284+0000 7f24d8ff9700 1 -- 172.21.15.132:0/1751406949 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 15) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f24cc00f070 con 0x7f24dc0703b0 2023-12-06T03:52:58.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.284+0000 7f24d8ff9700 1 --2- 172.21.15.132:0/1751406949 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f24c802ff00 0x7f24c80323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:52:58.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.284+0000 7f24d8ff9700 1 -- 172.21.15.132:0/1751406949 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f24cc0411a0 con 0x7f24dc0703b0 2023-12-06T03:52:58.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.284+0000 7f24db7fe700 1 --2- 172.21.15.132:0/1751406949 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f24c802ff00 0x7f24c80323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:52:58.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.284+0000 7f24db7fe700 1 --2- 172.21.15.132:0/1751406949 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f24c802ff00 0x7f24c80323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f24d00096a0 tx=0x7f24d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:52:58.288 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.284+0000 7f24e236c700 1 -- 172.21.15.132:0/1751406949 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24dc101e20 con 0x7f24dc0703b0 2023-12-06T03:52:58.290 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.288+0000 7f24d8ff9700 1 -- 172.21.15.132:0/1751406949 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f24cc016070 con 0x7f24dc0703b0 2023-12-06T03:52:58.444 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.442+0000 7f24e236c700 1 -- 172.21.15.132:0/1751406949 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f24dc10e670 con 0x7f24dc0703b0 2023-12-06T03:52:58.444 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.442+0000 7f24d8ff9700 1 -- 172.21.15.132:0/1751406949 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7f24cc010540 con 0x7f24dc0703b0 2023-12-06T03:52:58.444 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 8443 2023-12-06T03:52:58.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.444+0000 7f24e236c700 1 -- 172.21.15.132:0/1751406949 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f24c802ff00 msgr2=0x7f24c80323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:58.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.444+0000 7f24e236c700 1 --2- 172.21.15.132:0/1751406949 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f24c802ff00 0x7f24c80323c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f24d00096a0 tx=0x7f24d0006b40).stop 2023-12-06T03:52:58.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.444+0000 7f24e236c700 1 -- 172.21.15.132:0/1751406949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24dc0703b0 msgr2=0x7f24dc10cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:52:58.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.444+0000 7f24e236c700 1 --2- 172.21.15.132:0/1751406949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24dc0703b0 0x7f24dc10cd40 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f24cc00dfb0 tx=0x7f24cc006ee0).stop 2023-12-06T03:52:58.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.444+0000 7f24e236c700 1 -- 172.21.15.132:0/1751406949 shutdown_connections 2023-12-06T03:52:58.447 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.444+0000 7f24e236c700 1 --2- 172.21.15.132:0/1751406949 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f24c802ff00 0x7f24c80323c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:58.448 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.444+0000 7f24e236c700 1 --2- 172.21.15.132:0/1751406949 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24dc0703b0 0x7f24dc10cd40 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:52:58.448 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.444+0000 7f24e236c700 1 -- 172.21.15.132:0/1751406949 >> 172.21.15.132:0/1751406949 conn(0x7f24dc0f63c0 msgr2=0x7f24dc0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:52:58.448 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.444+0000 7f24e236c700 1 -- 172.21.15.132:0/1751406949 shutdown_connections 2023-12-06T03:52:58.448 INFO:teuthology.orchestra.run.smithi132.stderr:/usr/bin/ceph: 2023-12-06T03:52:58.444+0000 7f24e236c700 1 -- 172.21.15.132:0/1751406949 wait complete. 2023-12-06T03:52:58.792 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-enabled firewalld.service 2023-12-06T03:52:58.800 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: disabled 2023-12-06T03:52:58.801 INFO:teuthology.orchestra.run.smithi132.stderr:Running command: systemctl is-active firewalld.service 2023-12-06T03:52:58.807 INFO:teuthology.orchestra.run.smithi132.stderr:systemctl: inactive 2023-12-06T03:52:58.808 INFO:teuthology.orchestra.run.smithi132.stderr:firewalld.service is not enabled 2023-12-06T03:52:58.808 INFO:teuthology.orchestra.run.smithi132.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-12-06T03:52:58.811 INFO:teuthology.orchestra.run.smithi132.stderr:Ceph Dashboard is now available at: 2023-12-06T03:52:58.811 INFO:teuthology.orchestra.run.smithi132.stderr: 2023-12-06T03:52:58.811 INFO:teuthology.orchestra.run.smithi132.stderr: URL: https://smithi132.front.sepia.ceph.com:8443/ 2023-12-06T03:52:58.811 INFO:teuthology.orchestra.run.smithi132.stderr: User: admin 2023-12-06T03:52:58.811 INFO:teuthology.orchestra.run.smithi132.stderr: Password: 9ispg4vgz2 2023-12-06T03:52:58.811 INFO:teuthology.orchestra.run.smithi132.stderr: 2023-12-06T03:52:58.812 INFO:teuthology.orchestra.run.smithi132.stderr:You can access the Ceph CLI with: 2023-12-06T03:52:58.812 INFO:teuthology.orchestra.run.smithi132.stderr: 2023-12-06T03:52:58.812 INFO:teuthology.orchestra.run.smithi132.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-12-06T03:52:58.812 INFO:teuthology.orchestra.run.smithi132.stderr: 2023-12-06T03:52:58.812 INFO:teuthology.orchestra.run.smithi132.stderr:Please consider enabling telemetry to help improve Ceph: 2023-12-06T03:52:58.812 INFO:teuthology.orchestra.run.smithi132.stderr: 2023-12-06T03:52:58.812 INFO:teuthology.orchestra.run.smithi132.stderr: ceph telemetry on 2023-12-06T03:52:58.812 INFO:teuthology.orchestra.run.smithi132.stderr: 2023-12-06T03:52:58.812 INFO:teuthology.orchestra.run.smithi132.stderr:For more information see: 2023-12-06T03:52:58.812 INFO:teuthology.orchestra.run.smithi132.stderr: 2023-12-06T03:52:58.812 INFO:teuthology.orchestra.run.smithi132.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-12-06T03:52:58.812 INFO:teuthology.orchestra.run.smithi132.stderr: 2023-12-06T03:52:58.812 INFO:teuthology.orchestra.run.smithi132.stderr:Bootstrap complete. 2023-12-06T03:52:58.812 INFO:teuthology.orchestra.run.smithi132.stderr:Releasing lock 139868735597032 on /run/cephadm/c121c6d8-93ea-11ee-95a2-87774f69a715.lock 2023-12-06T03:52:58.813 INFO:teuthology.orchestra.run.smithi132.stderr:Lock 139868735597032 released on /run/cephadm/c121c6d8-93ea-11ee-95a2-87774f69a715.lock 2023-12-06T03:52:58.830 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:58 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:57.531842+0000 mon.smithi132 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' 2023-12-06T03:52:58.830 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:52:58 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:52:58.444056+0000 mon.smithi132 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.132:0/1751406949' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-12-06T03:52:58.844 INFO:tasks.cephadm:Fetching config... 2023-12-06T03:52:58.844 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T03:52:58.844 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-12-06T03:52:58.862 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-12-06T03:52:58.863 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T03:52:58.863 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-12-06T03:52:58.920 INFO:tasks.cephadm:Fetching mon keyring... 2023-12-06T03:52:58.920 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T03:52:58.920 DEBUG:teuthology.orchestra.run.smithi132:> sudo dd if=/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/mon.smithi132/keyring of=/dev/stdout 2023-12-06T03:52:58.990 INFO:tasks.cephadm:Fetching pub ssh key... 2023-12-06T03:52:58.990 DEBUG:teuthology.orchestra.run.smithi132:> set -ex 2023-12-06T03:52:58.990 DEBUG:teuthology.orchestra.run.smithi132:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-12-06T03:52:59.049 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-12-06T03:52:59.050 DEBUG:teuthology.orchestra.run.smithi132:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmtWJq0ND9ami/WEfjlFjk0GN9nrC6a7gbEGPXxm284p1GEkAT8/urB9hxmKS1PO0m59Fl/ePaPd0G3skDGMiJBg7axICw6Du0qD4ffgVWm9IxPc5pNOU88wj30TL5ksttzh+Ju8/hUva2mRgwFAZOuS0yfH9wwy5TtHHQ5uFBt8xnghWV7oRMGA8U236rtvR9RLPxfky6bhiPqyow3JBX2CIjzT8uD68n3VqE8w4GdZCeN59BV1i+dl5ihdioUP6XzlTg8DyncNgZLzle0s4XuCPt2xYxucfEB9/B2QoMJHQOiB4kRcniP8uFIqLiobF7ahL6ge96YiSQNKGWrZyHk1Y9/L3XlBv+MfRURM+9MQ8vFQZes3UwfMEoHwnHs0GYpuLgWM1Gevuh3XTCLrhIq2vHzvYVV0zXjwooHA2ryvtR2IEEjNDnmEnPP95l0gb1cfqmbnb/9JGh6GlBX6QmJDnxcHzQF5/v6/N4P3U+VMTD8yeNjh/toMVOFBe5frE= ceph-c121c6d8-93ea-11ee-95a2-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-06T03:52:59.127 INFO:teuthology.orchestra.run.smithi132.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmtWJq0ND9ami/WEfjlFjk0GN9nrC6a7gbEGPXxm284p1GEkAT8/urB9hxmKS1PO0m59Fl/ePaPd0G3skDGMiJBg7axICw6Du0qD4ffgVWm9IxPc5pNOU88wj30TL5ksttzh+Ju8/hUva2mRgwFAZOuS0yfH9wwy5TtHHQ5uFBt8xnghWV7oRMGA8U236rtvR9RLPxfky6bhiPqyow3JBX2CIjzT8uD68n3VqE8w4GdZCeN59BV1i+dl5ihdioUP6XzlTg8DyncNgZLzle0s4XuCPt2xYxucfEB9/B2QoMJHQOiB4kRcniP8uFIqLiobF7ahL6ge96YiSQNKGWrZyHk1Y9/L3XlBv+MfRURM+9MQ8vFQZes3UwfMEoHwnHs0GYpuLgWM1Gevuh3XTCLrhIq2vHzvYVV0zXjwooHA2ryvtR2IEEjNDnmEnPP95l0gb1cfqmbnb/9JGh6GlBX6QmJDnxcHzQF5/v6/N4P3U+VMTD8yeNjh/toMVOFBe5frE= ceph-c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:52:59.138 DEBUG:teuthology.orchestra.run.smithi152:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmtWJq0ND9ami/WEfjlFjk0GN9nrC6a7gbEGPXxm284p1GEkAT8/urB9hxmKS1PO0m59Fl/ePaPd0G3skDGMiJBg7axICw6Du0qD4ffgVWm9IxPc5pNOU88wj30TL5ksttzh+Ju8/hUva2mRgwFAZOuS0yfH9wwy5TtHHQ5uFBt8xnghWV7oRMGA8U236rtvR9RLPxfky6bhiPqyow3JBX2CIjzT8uD68n3VqE8w4GdZCeN59BV1i+dl5ihdioUP6XzlTg8DyncNgZLzle0s4XuCPt2xYxucfEB9/B2QoMJHQOiB4kRcniP8uFIqLiobF7ahL6ge96YiSQNKGWrZyHk1Y9/L3XlBv+MfRURM+9MQ8vFQZes3UwfMEoHwnHs0GYpuLgWM1Gevuh3XTCLrhIq2vHzvYVV0zXjwooHA2ryvtR2IEEjNDnmEnPP95l0gb1cfqmbnb/9JGh6GlBX6QmJDnxcHzQF5/v6/N4P3U+VMTD8yeNjh/toMVOFBe5frE= ceph-c121c6d8-93ea-11ee-95a2-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-06T03:52:59.183 INFO:teuthology.orchestra.run.smithi152.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDmtWJq0ND9ami/WEfjlFjk0GN9nrC6a7gbEGPXxm284p1GEkAT8/urB9hxmKS1PO0m59Fl/ePaPd0G3skDGMiJBg7axICw6Du0qD4ffgVWm9IxPc5pNOU88wj30TL5ksttzh+Ju8/hUva2mRgwFAZOuS0yfH9wwy5TtHHQ5uFBt8xnghWV7oRMGA8U236rtvR9RLPxfky6bhiPqyow3JBX2CIjzT8uD68n3VqE8w4GdZCeN59BV1i+dl5ihdioUP6XzlTg8DyncNgZLzle0s4XuCPt2xYxucfEB9/B2QoMJHQOiB4kRcniP8uFIqLiobF7ahL6ge96YiSQNKGWrZyHk1Y9/L3XlBv+MfRURM+9MQ8vFQZes3UwfMEoHwnHs0GYpuLgWM1Gevuh3XTCLrhIq2vHzvYVV0zXjwooHA2ryvtR2IEEjNDnmEnPP95l0gb1cfqmbnb/9JGh6GlBX6QmJDnxcHzQF5/v6/N4P3U+VMTD8yeNjh/toMVOFBe5frE= ceph-c121c6d8-93ea-11ee-95a2-87774f69a715 2023-12-06T03:52:59.195 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-12-06T03:53:00.576 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.573+0000 7f37c4338700 1 -- 172.21.15.132:0/930063771 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37bc0f6510 msgr2=0x7f37bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:00.576 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.573+0000 7f37c4338700 1 --2- 172.21.15.132:0/930063771 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37bc0f6510 0x7f37bc0f6930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f37ac009fb0 tx=0x7f37ac009b30).stop 2023-12-06T03:53:00.577 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.573+0000 7f37c4338700 1 -- 172.21.15.132:0/930063771 shutdown_connections 2023-12-06T03:53:00.577 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.573+0000 7f37c4338700 1 --2- 172.21.15.132:0/930063771 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37bc0f6510 0x7f37bc0f6930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:00.577 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.573+0000 7f37c4338700 1 -- 172.21.15.132:0/930063771 >> 172.21.15.132:0/930063771 conn(0x7f37bc0f1c70 msgr2=0x7f37bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:00.577 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.573+0000 7f37c4338700 1 -- 172.21.15.132:0/930063771 shutdown_connections 2023-12-06T03:53:00.577 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.573+0000 7f37c4338700 1 -- 172.21.15.132:0/930063771 wait complete. 2023-12-06T03:53:00.577 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.573+0000 7f37c4338700 1 Processor -- start 2023-12-06T03:53:00.577 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.574+0000 7f37c4338700 1 -- start start 2023-12-06T03:53:00.577 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.574+0000 7f37c4338700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37bc0f6510 0x7f37bc106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:00.577 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.574+0000 7f37c4338700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37bc106bf0 con 0x7f37bc0f6510 2023-12-06T03:53:00.577 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.574+0000 7f37c20d4700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37bc0f6510 0x7f37bc106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:00.577 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.574+0000 7f37c20d4700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37bc0f6510 0x7f37bc106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:55206/0 (socket says 172.21.15.132:55206) 2023-12-06T03:53:00.577 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.574+0000 7f37c20d4700 1 -- 172.21.15.132:0/3642585314 learned_addr learned my addr 172.21.15.132:0/3642585314 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:53:00.577 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.574+0000 7f37c20d4700 1 -- 172.21.15.132:0/3642585314 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37ac005040 con 0x7f37bc0f6510 2023-12-06T03:53:00.577 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.574+0000 7f37c20d4700 1 --2- 172.21.15.132:0/3642585314 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37bc0f6510 0x7f37bc106700 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f37ac013510 tx=0x7f37ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:00.578 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.574+0000 7f37b2ffd700 1 -- 172.21.15.132:0/3642585314 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37ac009320 con 0x7f37bc0f6510 2023-12-06T03:53:00.578 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.574+0000 7f37b2ffd700 1 -- 172.21.15.132:0/3642585314 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37ac013f00 con 0x7f37bc0f6510 2023-12-06T03:53:00.578 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.574+0000 7f37c4338700 1 -- 172.21.15.132:0/3642585314 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37bc106df0 con 0x7f37bc0f6510 2023-12-06T03:53:00.578 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.574+0000 7f37b2ffd700 1 -- 172.21.15.132:0/3642585314 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37ac014d90 con 0x7f37bc0f6510 2023-12-06T03:53:00.578 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.574+0000 7f37c4338700 1 -- 172.21.15.132:0/3642585314 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37bc107280 con 0x7f37bc0f6510 2023-12-06T03:53:00.579 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.575+0000 7f37b2ffd700 1 -- 172.21.15.132:0/3642585314 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37ac0263a0 con 0x7f37bc0f6510 2023-12-06T03:53:00.579 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.575+0000 7f37b2ffd700 1 --2- 172.21.15.132:0/3642585314 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f37a802ffb0 0x7f37a8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:00.579 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.575+0000 7f37b2ffd700 1 -- 172.21.15.132:0/3642585314 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37ac040f90 con 0x7f37bc0f6510 2023-12-06T03:53:00.579 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.575+0000 7f37c18d3700 1 --2- 172.21.15.132:0/3642585314 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f37a802ffb0 0x7f37a8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:00.579 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.576+0000 7f37c4338700 1 -- 172.21.15.132:0/3642585314 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37bc059100 con 0x7f37bc0f6510 2023-12-06T03:53:00.579 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.576+0000 7f37c18d3700 1 --2- 172.21.15.132:0/3642585314 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f37a802ffb0 0x7f37a8032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f37b80096a0 tx=0x7f37b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:00.582 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.580+0000 7f37b2ffd700 1 -- 172.21.15.132:0/3642585314 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37ac012920 con 0x7f37bc0f6510 2023-12-06T03:53:00.742 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.740+0000 7f37c4338700 1 -- 172.21.15.132:0/3642585314 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f37bc045ad0 con 0x7f37bc0f6510 2023-12-06T03:53:00.745 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.743+0000 7f37b2ffd700 1 -- 172.21.15.132:0/3642585314 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f37ac02b020 con 0x7f37bc0f6510 2023-12-06T03:53:00.747 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.745+0000 7f37c4338700 1 -- 172.21.15.132:0/3642585314 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f37a802ffb0 msgr2=0x7f37a8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:00.747 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.745+0000 7f37c4338700 1 --2- 172.21.15.132:0/3642585314 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f37a802ffb0 0x7f37a8032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f37b80096a0 tx=0x7f37b8006b40).stop 2023-12-06T03:53:00.747 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.745+0000 7f37c4338700 1 -- 172.21.15.132:0/3642585314 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37bc0f6510 msgr2=0x7f37bc106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:00.747 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.745+0000 7f37c4338700 1 --2- 172.21.15.132:0/3642585314 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37bc0f6510 0x7f37bc106700 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f37ac013510 tx=0x7f37ac00b040).stop 2023-12-06T03:53:00.747 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.745+0000 7f37c4338700 1 -- 172.21.15.132:0/3642585314 shutdown_connections 2023-12-06T03:53:00.747 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.745+0000 7f37c4338700 1 --2- 172.21.15.132:0/3642585314 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f37a802ffb0 0x7f37a8032470 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:00.747 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.745+0000 7f37c4338700 1 --2- 172.21.15.132:0/3642585314 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37bc0f6510 0x7f37bc106700 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:00.747 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.745+0000 7f37c4338700 1 -- 172.21.15.132:0/3642585314 >> 172.21.15.132:0/3642585314 conn(0x7f37bc0f1c70 msgr2=0x7f37bc0f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:00.747 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.745+0000 7f37c4338700 1 -- 172.21.15.132:0/3642585314 shutdown_connections 2023-12-06T03:53:00.748 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:00.745+0000 7f37c4338700 1 -- 172.21.15.132:0/3642585314 wait complete. 2023-12-06T03:53:01.412 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi152 2023-12-06T03:53:01.412 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-06T03:53:01.412 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.conf 2023-12-06T03:53:01.434 DEBUG:teuthology.orchestra.run.smithi152:> set -ex 2023-12-06T03:53:01.435 DEBUG:teuthology.orchestra.run.smithi152:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-06T03:53:01.494 INFO:tasks.cephadm:Adding host smithi152 to orchestrator... 2023-12-06T03:53:01.495 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph orch host add smithi152 2023-12-06T03:53:01.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:01 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:00.391081+0000 mon.smithi132 (mon.0) 112 : cluster [DBG] mgrmap e16: smithi132.gokxja(active, since 6s) 2023-12-06T03:53:01.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:01 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:00.744917+0000 mon.smithi132 (mon.0) 113 : audit [INF] from='client.? 172.21.15.132:0/3642585314' entity='client.admin' 2023-12-06T03:53:04.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:04 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:03.569314+0000 mon.smithi132 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' 2023-12-06T03:53:04.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:04 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:03.845581+0000 mon.smithi132 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' 2023-12-06T03:53:04.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:04 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:53:03.851785+0000 mgr.smithi132.gokxja (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi132 on smithi132 2023-12-06T03:53:15.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:15 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:14.382513+0000 mgr.smithi132.gokxja (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:17.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:17 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:16.382831+0000 mgr.smithi132.gokxja (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:19.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:19 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:18.383194+0000 mgr.smithi132.gokxja (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:21.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:20.383614+0000 mgr.smithi132.gokxja (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:22.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.186+0000 7f05029f1700 1 -- 172.21.15.152:0/539831281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04fc0f51a0 msgr2=0x7f04fc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:22.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.186+0000 7f05029f1700 1 --2- 172.21.15.152:0/539831281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04fc0f51a0 0x7f04fc0f55c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f04ec009fb0 tx=0x7f04ec009b30).stop 2023-12-06T03:53:22.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.186+0000 7f05029f1700 1 -- 172.21.15.152:0/539831281 shutdown_connections 2023-12-06T03:53:22.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.186+0000 7f05029f1700 1 --2- 172.21.15.152:0/539831281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04fc0f51a0 0x7f04fc0f55c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:22.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.186+0000 7f05029f1700 1 -- 172.21.15.152:0/539831281 >> 172.21.15.152:0/539831281 conn(0x7f04fc0f09a0 msgr2=0x7f04fc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:22.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.186+0000 7f05029f1700 1 -- 172.21.15.152:0/539831281 shutdown_connections 2023-12-06T03:53:22.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.186+0000 7f05029f1700 1 -- 172.21.15.152:0/539831281 wait complete. 2023-12-06T03:53:22.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.187+0000 7f05029f1700 1 Processor -- start 2023-12-06T03:53:22.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.187+0000 7f05029f1700 1 -- start start 2023-12-06T03:53:22.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.187+0000 7f05029f1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04fc0f51a0 0x7f04fc1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:22.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.187+0000 7f05029f1700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f04fc060220 con 0x7f04fc0f51a0 2023-12-06T03:53:22.189 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.187+0000 7f05019ef700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04fc0f51a0 0x7f04fc1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:22.189 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.187+0000 7f05019ef700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04fc0f51a0 0x7f04fc1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:41756/0 (socket says 172.21.15.152:41756) 2023-12-06T03:53:22.189 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.187+0000 7f05019ef700 1 -- 172.21.15.152:0/2647709712 learned_addr learned my addr 172.21.15.152:0/2647709712 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:53:22.190 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.187+0000 7f05019ef700 1 -- 172.21.15.152:0/2647709712 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04ec005040 con 0x7f04fc0f51a0 2023-12-06T03:53:22.190 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.188+0000 7f05019ef700 1 --2- 172.21.15.152:0/2647709712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04fc0f51a0 0x7f04fc1091d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f04ec013510 tx=0x7f04ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:22.190 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.188+0000 7f04f27fc700 1 -- 172.21.15.152:0/2647709712 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04ec009320 con 0x7f04fc0f51a0 2023-12-06T03:53:22.190 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.188+0000 7f05029f1700 1 -- 172.21.15.152:0/2647709712 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04fc060420 con 0x7f04fc0f51a0 2023-12-06T03:53:22.190 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.188+0000 7f04f27fc700 1 -- 172.21.15.152:0/2647709712 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f04ec013f00 con 0x7f04fc0f51a0 2023-12-06T03:53:22.190 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.188+0000 7f04f27fc700 1 -- 172.21.15.152:0/2647709712 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f04ec014dc0 con 0x7f04fc0f51a0 2023-12-06T03:53:22.191 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.188+0000 7f05029f1700 1 -- 172.21.15.152:0/2647709712 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04fc109880 con 0x7f04fc0f51a0 2023-12-06T03:53:22.191 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.189+0000 7f04f27fc700 1 -- 172.21.15.152:0/2647709712 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04ec0095f0 con 0x7f04fc0f51a0 2023-12-06T03:53:22.191 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.189+0000 7f05029f1700 1 -- 172.21.15.152:0/2647709712 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04fc059100 con 0x7f04fc0f51a0 2023-12-06T03:53:22.191 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.189+0000 7f04f27fc700 1 --2- 172.21.15.152:0/2647709712 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f04e802ffb0 0x7f04e8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:22.191 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.189+0000 7f04f27fc700 1 -- 172.21.15.152:0/2647709712 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f04ec040f90 con 0x7f04fc0f51a0 2023-12-06T03:53:22.191 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.189+0000 7f05011ee700 1 --2- 172.21.15.152:0/2647709712 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f04e802ffb0 0x7f04e8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:22.191 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.189+0000 7f05011ee700 1 --2- 172.21.15.152:0/2647709712 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f04e802ffb0 0x7f04e8032470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f04f80096a0 tx=0x7f04f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:22.195 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.193+0000 7f04f27fc700 1 -- 172.21.15.152:0/2647709712 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04ec014420 con 0x7f04fc0f51a0 2023-12-06T03:53:22.365 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.364+0000 7f05029f1700 1 -- 172.21.15.152:0/2647709712 --> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}) v1 -- 0x7f04fc0f1b70 con 0x7f04e802ffb0 2023-12-06T03:53:22.963 INFO:teuthology.orchestra.run.smithi152.stdout:Added host 'smithi152' 2023-12-06T03:53:22.963 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.961+0000 7f04f27fc700 1 -- 172.21.15.152:0/2647709712 <== mgr.14162 v2:172.21.15.132:6800/1042262986 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f04fc0f1b70 con 0x7f04e802ffb0 2023-12-06T03:53:22.964 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.963+0000 7f05029f1700 1 -- 172.21.15.152:0/2647709712 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f04e802ffb0 msgr2=0x7f04e8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:22.964 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.963+0000 7f05029f1700 1 --2- 172.21.15.152:0/2647709712 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f04e802ffb0 0x7f04e8032470 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f04f80096a0 tx=0x7f04f8006b40).stop 2023-12-06T03:53:22.964 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.963+0000 7f05029f1700 1 -- 172.21.15.152:0/2647709712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04fc0f51a0 msgr2=0x7f04fc1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:22.964 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.963+0000 7f05029f1700 1 --2- 172.21.15.152:0/2647709712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04fc0f51a0 0x7f04fc1091d0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f04ec013510 tx=0x7f04ec00b040).stop 2023-12-06T03:53:22.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.963+0000 7f05029f1700 1 -- 172.21.15.152:0/2647709712 shutdown_connections 2023-12-06T03:53:22.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.963+0000 7f05029f1700 1 --2- 172.21.15.152:0/2647709712 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f04e802ffb0 0x7f04e8032470 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:22.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.963+0000 7f05029f1700 1 --2- 172.21.15.152:0/2647709712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04fc0f51a0 0x7f04fc1091d0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:22.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.963+0000 7f05029f1700 1 -- 172.21.15.152:0/2647709712 >> 172.21.15.152:0/2647709712 conn(0x7f04fc0f09a0 msgr2=0x7f04fc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:22.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.963+0000 7f05029f1700 1 -- 172.21.15.152:0/2647709712 shutdown_connections 2023-12-06T03:53:22.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:22.963+0000 7f05029f1700 1 -- 172.21.15.152:0/2647709712 wait complete. 2023-12-06T03:53:23.313 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph orch host ls --format=json 2023-12-06T03:53:24.356 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:22.365742+0000 mgr.smithi132.gokxja (mgr.14162) 10 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi152", "target": ["mon-mgr", ""]}]: dispatch 2023-12-06T03:53:24.356 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:22.383970+0000 mgr.smithi132.gokxja (mgr.14162) 11 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:24.357 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:22.960790+0000 mon.smithi132 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' 2023-12-06T03:53:24.357 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:53:22.961196+0000 mgr.smithi132.gokxja (mgr.14162) 12 : cephadm [INF] Added host smithi152 2023-12-06T03:53:24.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.410+0000 7f3daea44700 1 -- 172.21.15.152:0/1189577119 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80f5410 msgr2=0x7f3da80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:24.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.410+0000 7f3daea44700 1 --2- 172.21.15.152:0/1189577119 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80f5410 0x7f3da80f5830 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f3d98009960 tx=0x7f3d98004d10).stop 2023-12-06T03:53:24.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.410+0000 7f3daea44700 1 -- 172.21.15.152:0/1189577119 shutdown_connections 2023-12-06T03:53:24.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.410+0000 7f3daea44700 1 --2- 172.21.15.152:0/1189577119 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80f5410 0x7f3da80f5830 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:24.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.410+0000 7f3daea44700 1 -- 172.21.15.152:0/1189577119 >> 172.21.15.152:0/1189577119 conn(0x7f3da80f0b70 msgr2=0x7f3da80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:24.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.410+0000 7f3daea44700 1 -- 172.21.15.152:0/1189577119 shutdown_connections 2023-12-06T03:53:24.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.410+0000 7f3daea44700 1 -- 172.21.15.152:0/1189577119 wait complete. 2023-12-06T03:53:24.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.410+0000 7f3daea44700 1 Processor -- start 2023-12-06T03:53:24.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.410+0000 7f3daea44700 1 -- start start 2023-12-06T03:53:24.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.410+0000 7f3daea44700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80f5410 0x7f3da8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:24.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.410+0000 7f3daea44700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3da8060240 con 0x7f3da80f5410 2023-12-06T03:53:24.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.411+0000 7f3da7fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80f5410 0x7f3da8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:24.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.411+0000 7f3da7fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80f5410 0x7f3da8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:41778/0 (socket says 172.21.15.152:41778) 2023-12-06T03:53:24.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.411+0000 7f3da7fff700 1 -- 172.21.15.152:0/2819265501 learned_addr learned my addr 172.21.15.152:0/2819265501 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:53:24.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.411+0000 7f3da7fff700 1 -- 172.21.15.152:0/2819265501 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d98009640 con 0x7f3da80f5410 2023-12-06T03:53:24.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.411+0000 7f3da7fff700 1 --2- 172.21.15.152:0/2819265501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80f5410 0x7f3da8109430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f3d98004b70 tx=0x7f3d98004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:24.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.411+0000 7f3da57fa700 1 -- 172.21.15.152:0/2819265501 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d98018070 con 0x7f3da80f5410 2023-12-06T03:53:24.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.411+0000 7f3daea44700 1 -- 172.21.15.152:0/2819265501 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3da8060440 con 0x7f3da80f5410 2023-12-06T03:53:24.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.411+0000 7f3da57fa700 1 -- 172.21.15.152:0/2819265501 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d9800b870 con 0x7f3da80f5410 2023-12-06T03:53:24.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.411+0000 7f3daea44700 1 -- 172.21.15.152:0/2819265501 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3da8109ae0 con 0x7f3da80f5410 2023-12-06T03:53:24.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.411+0000 7f3da57fa700 1 -- 172.21.15.152:0/2819265501 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d9801d580 con 0x7f3da80f5410 2023-12-06T03:53:24.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.412+0000 7f3da57fa700 1 -- 172.21.15.152:0/2819265501 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d98013630 con 0x7f3da80f5410 2023-12-06T03:53:24.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.412+0000 7f3da57fa700 1 --2- 172.21.15.152:0/2819265501 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3d9402ff60 0x7f3d94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:24.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.412+0000 7f3daea44700 1 -- 172.21.15.152:0/2819265501 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3da8045ad0 con 0x7f3da80f5410 2023-12-06T03:53:24.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.412+0000 7f3da57fa700 1 -- 172.21.15.152:0/2819265501 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d98041230 con 0x7f3da80f5410 2023-12-06T03:53:24.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.412+0000 7f3d9ffff700 1 --2- 172.21.15.152:0/2819265501 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3d9402ff60 0x7f3d94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:24.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.413+0000 7f3d9ffff700 1 --2- 172.21.15.152:0/2819265501 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3d9402ff60 0x7f3d94032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f3d900096a0 tx=0x7f3d90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:24.418 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.417+0000 7f3da57fa700 1 -- 172.21.15.152:0/2819265501 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d9801c7f0 con 0x7f3da80f5410 2023-12-06T03:53:24.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.588+0000 7f3daea44700 1 -- 172.21.15.152:0/2819265501 --> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3da80f1d70 con 0x7f3d9402ff60 2023-12-06T03:53:24.590 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.590+0000 7f3da57fa700 1 -- 172.21.15.152:0/2819265501 <== mgr.14162 v2:172.21.15.132:6800/1042262986 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f3da80f1d70 con 0x7f3d9402ff60 2023-12-06T03:53:24.590 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:53:24.591 INFO:teuthology.orchestra.run.smithi152.stdout:[{"addr": "172.21.15.132", "hostname": "smithi132", "labels": [], "status": ""}, {"addr": "smithi152", "hostname": "smithi152", "labels": [], "status": ""}] 2023-12-06T03:53:24.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.591+0000 7f3daea44700 1 -- 172.21.15.152:0/2819265501 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3d9402ff60 msgr2=0x7f3d94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:24.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.591+0000 7f3daea44700 1 --2- 172.21.15.152:0/2819265501 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3d9402ff60 0x7f3d94032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f3d900096a0 tx=0x7f3d90006b40).stop 2023-12-06T03:53:24.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.591+0000 7f3daea44700 1 -- 172.21.15.152:0/2819265501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80f5410 msgr2=0x7f3da8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:24.593 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.591+0000 7f3daea44700 1 --2- 172.21.15.152:0/2819265501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80f5410 0x7f3da8109430 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f3d98004b70 tx=0x7f3d98004d10).stop 2023-12-06T03:53:24.593 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.591+0000 7f3daea44700 1 -- 172.21.15.152:0/2819265501 shutdown_connections 2023-12-06T03:53:24.593 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.591+0000 7f3daea44700 1 --2- 172.21.15.152:0/2819265501 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3d9402ff60 0x7f3d94032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:24.593 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.591+0000 7f3daea44700 1 --2- 172.21.15.152:0/2819265501 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3da80f5410 0x7f3da8109430 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:24.593 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.591+0000 7f3daea44700 1 -- 172.21.15.152:0/2819265501 >> 172.21.15.152:0/2819265501 conn(0x7f3da80f0b70 msgr2=0x7f3da80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:24.593 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.591+0000 7f3daea44700 1 -- 172.21.15.152:0/2819265501 shutdown_connections 2023-12-06T03:53:24.593 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:24.591+0000 7f3daea44700 1 -- 172.21.15.152:0/2819265501 wait complete. 2023-12-06T03:53:24.987 INFO:tasks.cephadm:Setting crush tunables to default 2023-12-06T03:53:24.987 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph osd crush tunables default 2023-12-06T03:53:26.089 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.085+0000 7feb9acf7700 1 -- 172.21.15.132:0/2602352039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f62a0 msgr2=0x7feb940f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:26.089 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.085+0000 7feb9acf7700 1 --2- 172.21.15.132:0/2602352039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f62a0 0x7feb940f66c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7feb84009fb0 tx=0x7feb84009b30).stop 2023-12-06T03:53:26.089 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.085+0000 7feb9acf7700 1 -- 172.21.15.132:0/2602352039 shutdown_connections 2023-12-06T03:53:26.089 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.085+0000 7feb9acf7700 1 --2- 172.21.15.132:0/2602352039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f62a0 0x7feb940f66c0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:26.089 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.085+0000 7feb9acf7700 1 -- 172.21.15.132:0/2602352039 >> 172.21.15.132:0/2602352039 conn(0x7feb940f1aa0 msgr2=0x7feb940f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:26.089 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.085+0000 7feb9acf7700 1 -- 172.21.15.132:0/2602352039 shutdown_connections 2023-12-06T03:53:26.089 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.085+0000 7feb9acf7700 1 -- 172.21.15.132:0/2602352039 wait complete. 2023-12-06T03:53:26.089 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.086+0000 7feb9acf7700 1 Processor -- start 2023-12-06T03:53:26.089 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.086+0000 7feb9acf7700 1 -- start start 2023-12-06T03:53:26.089 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.086+0000 7feb9acf7700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f62a0 0x7feb941064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:26.090 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.086+0000 7feb9acf7700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb941069a0 con 0x7feb940f62a0 2023-12-06T03:53:26.090 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.086+0000 7feb99cf5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f62a0 0x7feb941064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:26.090 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.086+0000 7feb99cf5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f62a0 0x7feb941064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.132:46494/0 (socket says 172.21.15.132:46494) 2023-12-06T03:53:26.090 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.086+0000 7feb99cf5700 1 -- 172.21.15.132:0/3348822169 learned_addr learned my addr 172.21.15.132:0/3348822169 (peer_addr_for_me v2:172.21.15.132:0/0) 2023-12-06T03:53:26.090 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.086+0000 7feb99cf5700 1 -- 172.21.15.132:0/3348822169 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb84005040 con 0x7feb940f62a0 2023-12-06T03:53:26.090 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.086+0000 7feb99cf5700 1 --2- 172.21.15.132:0/3348822169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f62a0 0x7feb941064b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7feb84013510 tx=0x7feb8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:26.090 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.087+0000 7feb8a7fc700 1 -- 172.21.15.132:0/3348822169 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb84009320 con 0x7feb940f62a0 2023-12-06T03:53:26.090 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.087+0000 7feb8a7fc700 1 -- 172.21.15.132:0/3348822169 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb84013f00 con 0x7feb940f62a0 2023-12-06T03:53:26.090 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.087+0000 7feb9acf7700 1 -- 172.21.15.132:0/3348822169 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb94106ba0 con 0x7feb940f62a0 2023-12-06T03:53:26.090 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.087+0000 7feb8a7fc700 1 -- 172.21.15.132:0/3348822169 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb84014dc0 con 0x7feb940f62a0 2023-12-06T03:53:26.090 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.087+0000 7feb9acf7700 1 -- 172.21.15.132:0/3348822169 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb94107030 con 0x7feb940f62a0 2023-12-06T03:53:26.091 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.087+0000 7feb8a7fc700 1 -- 172.21.15.132:0/3348822169 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb8401d3d0 con 0x7feb940f62a0 2023-12-06T03:53:26.091 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.087+0000 7feb8a7fc700 1 --2- 172.21.15.132:0/3348822169 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb8002ff60 0x7feb80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:26.091 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.088+0000 7feb8a7fc700 1 -- 172.21.15.132:0/3348822169 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb840410e0 con 0x7feb940f62a0 2023-12-06T03:53:26.091 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.088+0000 7feb9acf7700 1 -- 172.21.15.132:0/3348822169 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb94045ad0 con 0x7feb940f62a0 2023-12-06T03:53:26.091 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.088+0000 7feb994f4700 1 --2- 172.21.15.132:0/3348822169 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb8002ff60 0x7feb80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:26.091 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.088+0000 7feb994f4700 1 --2- 172.21.15.132:0/3348822169 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb8002ff60 0x7feb80032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7feb900096a0 tx=0x7feb90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:26.093 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.092+0000 7feb8a7fc700 1 -- 172.21.15.132:0/3348822169 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb84012900 con 0x7feb940f62a0 2023-12-06T03:53:26.246 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.244+0000 7feb9acf7700 1 -- 172.21.15.132:0/3348822169 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7feb94107d50 con 0x7feb940f62a0 2023-12-06T03:53:26.356 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:24.384369+0000 mgr.smithi132.gokxja (mgr.14162) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:26.356 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:24.589449+0000 mgr.smithi132.gokxja (mgr.14162) 14 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-06T03:53:26.965 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.963+0000 7feb8a7fc700 1 -- 172.21.15.132:0/3348822169 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7feb8402b030 con 0x7feb940f62a0 2023-12-06T03:53:26.968 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.966+0000 7feb9acf7700 1 -- 172.21.15.132:0/3348822169 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb8002ff60 msgr2=0x7feb80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:26.968 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.966+0000 7feb9acf7700 1 --2- 172.21.15.132:0/3348822169 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb8002ff60 0x7feb80032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7feb900096a0 tx=0x7feb90006b40).stop 2023-12-06T03:53:26.968 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.966+0000 7feb9acf7700 1 -- 172.21.15.132:0/3348822169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f62a0 msgr2=0x7feb941064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:26.968 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.966+0000 7feb9acf7700 1 --2- 172.21.15.132:0/3348822169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f62a0 0x7feb941064b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7feb84013510 tx=0x7feb8400b040).stop 2023-12-06T03:53:26.968 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.966+0000 7feb9acf7700 1 -- 172.21.15.132:0/3348822169 shutdown_connections 2023-12-06T03:53:26.968 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.966+0000 7feb9acf7700 1 --2- 172.21.15.132:0/3348822169 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb8002ff60 0x7feb80032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:26.968 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.966+0000 7feb9acf7700 1 --2- 172.21.15.132:0/3348822169 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f62a0 0x7feb941064b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:26.968 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.966+0000 7feb9acf7700 1 -- 172.21.15.132:0/3348822169 >> 172.21.15.132:0/3348822169 conn(0x7feb940f1aa0 msgr2=0x7feb940f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:26.968 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.966+0000 7feb9acf7700 1 -- 172.21.15.132:0/3348822169 shutdown_connections 2023-12-06T03:53:26.968 INFO:teuthology.orchestra.run.smithi132.stderr:2023-12-06T03:53:26.966+0000 7feb9acf7700 1 -- 172.21.15.132:0/3348822169 wait complete. 2023-12-06T03:53:26.969 INFO:teuthology.orchestra.run.smithi132.stderr:adjusted tunables profile to default 2023-12-06T03:53:27.232 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:26 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:26.245808+0000 mon.smithi132 (mon.0) 117 : audit [INF] from='client.? 172.21.15.132:0/3348822169' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-06T03:53:27.327 INFO:tasks.cephadm:Adding mon.smithi132 on smithi132 2023-12-06T03:53:27.327 INFO:tasks.cephadm:Adding mon.smithi152 on smithi152 2023-12-06T03:53:27.327 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph orch apply mon '2;smithi132:172.21.15.132=smithi132;smithi152:172.21.15.152=smithi152' 2023-12-06T03:53:28.357 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:26.384724+0000 mgr.smithi132.gokxja (mgr.14162) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:28.357 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:26.965226+0000 mon.smithi132 (mon.0) 118 : audit [INF] from='client.? 172.21.15.132:0/3348822169' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-06T03:53:28.357 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:26.965310+0000 mon.smithi132 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-12-06T03:53:28.401 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.397+0000 7f4157b4a700 1 -- 172.21.15.152:0/543894614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41500f5410 msgr2=0x7f41500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:28.401 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.397+0000 7f4157b4a700 1 --2- 172.21.15.152:0/543894614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41500f5410 0x7f41500f5830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f4140009fb0 tx=0x7f4140009b30).stop 2023-12-06T03:53:28.401 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.397+0000 7f4157b4a700 1 -- 172.21.15.152:0/543894614 shutdown_connections 2023-12-06T03:53:28.402 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.397+0000 7f4157b4a700 1 --2- 172.21.15.152:0/543894614 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41500f5410 0x7f41500f5830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:28.402 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.397+0000 7f4157b4a700 1 -- 172.21.15.152:0/543894614 >> 172.21.15.152:0/543894614 conn(0x7f41500f0b70 msgr2=0x7f41500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:28.402 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.397+0000 7f4157b4a700 1 -- 172.21.15.152:0/543894614 shutdown_connections 2023-12-06T03:53:28.402 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.397+0000 7f4157b4a700 1 -- 172.21.15.152:0/543894614 wait complete. 2023-12-06T03:53:28.402 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.398+0000 7f4157b4a700 1 Processor -- start 2023-12-06T03:53:28.402 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.398+0000 7f4157b4a700 1 -- start start 2023-12-06T03:53:28.402 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.398+0000 7f4157b4a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41500f5410 0x7f4150105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:28.402 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.398+0000 7f4157b4a700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4150105b20 con 0x7f41500f5410 2023-12-06T03:53:28.402 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.398+0000 7f41558e6700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41500f5410 0x7f4150105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:28.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.398+0000 7f41558e6700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41500f5410 0x7f4150105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:41790/0 (socket says 172.21.15.152:41790) 2023-12-06T03:53:28.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.398+0000 7f41558e6700 1 -- 172.21.15.152:0/1160505717 learned_addr learned my addr 172.21.15.152:0/1160505717 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:53:28.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.398+0000 7f41558e6700 1 -- 172.21.15.152:0/1160505717 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4140005040 con 0x7f41500f5410 2023-12-06T03:53:28.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.398+0000 7f41558e6700 1 --2- 172.21.15.152:0/1160505717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41500f5410 0x7f4150105630 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f4140014000 tx=0x7f414000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:28.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.398+0000 7f41467fc700 1 -- 172.21.15.152:0/1160505717 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f41400095f0 con 0x7f41500f5410 2023-12-06T03:53:28.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.399+0000 7f4157b4a700 1 -- 172.21.15.152:0/1160505717 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4150105d20 con 0x7f41500f5410 2023-12-06T03:53:28.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.399+0000 7f41467fc700 1 -- 172.21.15.152:0/1160505717 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4140007e70 con 0x7f41500f5410 2023-12-06T03:53:28.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.399+0000 7f41467fc700 1 -- 172.21.15.152:0/1160505717 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4140014ea0 con 0x7f41500f5410 2023-12-06T03:53:28.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.399+0000 7f4157b4a700 1 -- 172.21.15.152:0/1160505717 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41501061b0 con 0x7f41500f5410 2023-12-06T03:53:28.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.399+0000 7f41467fc700 1 -- 172.21.15.152:0/1160505717 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4140025070 con 0x7f41500f5410 2023-12-06T03:53:28.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.399+0000 7f4157b4a700 1 -- 172.21.15.152:0/1160505717 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4150045ad0 con 0x7f41500f5410 2023-12-06T03:53:28.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.400+0000 7f41467fc700 1 --2- 172.21.15.152:0/1160505717 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f413c02ff60 0x7f413c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:28.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.400+0000 7f41467fc700 1 -- 172.21.15.152:0/1160505717 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4140040970 con 0x7f41500f5410 2023-12-06T03:53:28.404 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.400+0000 7f41550e5700 1 --2- 172.21.15.152:0/1160505717 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f413c02ff60 0x7f413c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:28.404 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.400+0000 7f41550e5700 1 --2- 172.21.15.152:0/1160505717 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f413c02ff60 0x7f413c032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f414c0096a0 tx=0x7f414c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:28.405 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.404+0000 7f41467fc700 1 -- 172.21.15.152:0/1160505717 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41400079c0 con 0x7f41500f5410 2023-12-06T03:53:28.581 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.580+0000 7f4157b4a700 1 -- 172.21.15.152:0/1160505717 --> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi132:172.21.15.132=smithi132;smithi152:172.21.15.152=smithi152", "target": ["mon-mgr", ""]}) v1 -- 0x7f41500f1e20 con 0x7f413c02ff60 2023-12-06T03:53:28.587 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.586+0000 7f41467fc700 1 -- 172.21.15.152:0/1160505717 <== mgr.14162 v2:172.21.15.132:6800/1042262986 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f41500f1e20 con 0x7f413c02ff60 2023-12-06T03:53:28.587 INFO:teuthology.orchestra.run.smithi152.stdout:Scheduled mon update... 2023-12-06T03:53:28.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.588+0000 7f4157b4a700 1 -- 172.21.15.152:0/1160505717 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f413c02ff60 msgr2=0x7f413c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:28.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.588+0000 7f4157b4a700 1 --2- 172.21.15.152:0/1160505717 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f413c02ff60 0x7f413c032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f414c0096a0 tx=0x7f414c006b40).stop 2023-12-06T03:53:28.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.588+0000 7f4157b4a700 1 -- 172.21.15.152:0/1160505717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41500f5410 msgr2=0x7f4150105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:28.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.588+0000 7f4157b4a700 1 --2- 172.21.15.152:0/1160505717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41500f5410 0x7f4150105630 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f4140014000 tx=0x7f414000b040).stop 2023-12-06T03:53:28.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.588+0000 7f4157b4a700 1 -- 172.21.15.152:0/1160505717 shutdown_connections 2023-12-06T03:53:28.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.588+0000 7f4157b4a700 1 --2- 172.21.15.152:0/1160505717 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f413c02ff60 0x7f413c032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:28.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.588+0000 7f4157b4a700 1 --2- 172.21.15.152:0/1160505717 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f41500f5410 0x7f4150105630 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:28.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.588+0000 7f4157b4a700 1 -- 172.21.15.152:0/1160505717 >> 172.21.15.152:0/1160505717 conn(0x7f41500f0b70 msgr2=0x7f41500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:28.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.588+0000 7f4157b4a700 1 -- 172.21.15.152:0/1160505717 shutdown_connections 2023-12-06T03:53:28.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:28.588+0000 7f4157b4a700 1 -- 172.21.15.152:0/1160505717 wait complete. 2023-12-06T03:53:28.965 DEBUG:teuthology.orchestra.run.smithi152:mon.smithi152> sudo journalctl -f -n 0 -u ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi152.service 2023-12-06T03:53:28.967 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:53:28.968 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:53:28.996 INFO:journalctl@ceph.mon.smithi152.smithi152.stdout:-- Logs begin at Wed 2023-12-06 03:39:40 UTC. -- 2023-12-06T03:53:29.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:28.385005+0000 mgr.smithi132.gokxja (mgr.14162) 16 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:29.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:28.581767+0000 mgr.smithi132.gokxja (mgr.14162) 17 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi132:172.21.15.132=smithi132;smithi152:172.21.15.152=smithi152", "target": ["mon-mgr", ""]}]: dispatch 2023-12-06T03:53:29.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:53:28.583173+0000 mgr.smithi132.gokxja (mgr.14162) 18 : cephadm [INF] Saving service mon spec with placement smithi132:172.21.15.132=smithi132;smithi152:172.21.15.152=smithi152;count:2 2023-12-06T03:53:29.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:28.585971+0000 mon.smithi132 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' 2023-12-06T03:53:30.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.103+0000 7fcd2de19700 1 -- 172.21.15.152:0/333257589 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd280f5410 msgr2=0x7fcd280f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:30.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.103+0000 7fcd2de19700 1 --2- 172.21.15.152:0/333257589 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd280f5410 0x7fcd280f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fcd18009fb0 tx=0x7fcd18009b30).stop 2023-12-06T03:53:30.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.103+0000 7fcd2de19700 1 -- 172.21.15.152:0/333257589 shutdown_connections 2023-12-06T03:53:30.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.103+0000 7fcd2de19700 1 --2- 172.21.15.152:0/333257589 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd280f5410 0x7fcd280f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:30.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.103+0000 7fcd2de19700 1 -- 172.21.15.152:0/333257589 >> 172.21.15.152:0/333257589 conn(0x7fcd280f0b70 msgr2=0x7fcd280f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:30.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.103+0000 7fcd2de19700 1 -- 172.21.15.152:0/333257589 shutdown_connections 2023-12-06T03:53:30.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.103+0000 7fcd2de19700 1 -- 172.21.15.152:0/333257589 wait complete. 2023-12-06T03:53:30.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.103+0000 7fcd2de19700 1 Processor -- start 2023-12-06T03:53:30.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.103+0000 7fcd2de19700 1 -- start start 2023-12-06T03:53:30.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.104+0000 7fcd2de19700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd280f5410 0x7fcd28105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:30.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.104+0000 7fcd2de19700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd28105b20 con 0x7fcd280f5410 2023-12-06T03:53:30.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.104+0000 7fcd277fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd280f5410 0x7fcd28105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:30.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.104+0000 7fcd277fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd280f5410 0x7fcd28105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:57482/0 (socket says 172.21.15.152:57482) 2023-12-06T03:53:30.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.104+0000 7fcd277fe700 1 -- 172.21.15.152:0/3004489220 learned_addr learned my addr 172.21.15.152:0/3004489220 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:53:30.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.104+0000 7fcd277fe700 1 -- 172.21.15.152:0/3004489220 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd18005040 con 0x7fcd280f5410 2023-12-06T03:53:30.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.104+0000 7fcd277fe700 1 --2- 172.21.15.152:0/3004489220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd280f5410 0x7fcd28105630 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fcd18013510 tx=0x7fcd1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:30.108 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.104+0000 7fcd0ffff700 1 -- 172.21.15.152:0/3004489220 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd18009320 con 0x7fcd280f5410 2023-12-06T03:53:30.108 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.104+0000 7fcd0ffff700 1 -- 172.21.15.152:0/3004489220 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd18013f00 con 0x7fcd280f5410 2023-12-06T03:53:30.108 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.104+0000 7fcd2de19700 1 -- 172.21.15.152:0/3004489220 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd28105d20 con 0x7fcd280f5410 2023-12-06T03:53:30.109 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.104+0000 7fcd0ffff700 1 -- 172.21.15.152:0/3004489220 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd18014d90 con 0x7fcd280f5410 2023-12-06T03:53:30.109 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.104+0000 7fcd2de19700 1 -- 172.21.15.152:0/3004489220 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd281061b0 con 0x7fcd280f5410 2023-12-06T03:53:30.109 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.105+0000 7fcd0ffff700 1 -- 172.21.15.152:0/3004489220 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd18014420 con 0x7fcd280f5410 2023-12-06T03:53:30.109 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.105+0000 7fcd0ffff700 1 --2- 172.21.15.152:0/3004489220 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd1002ff60 0x7fcd10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:30.109 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.105+0000 7fcd0ffff700 1 -- 172.21.15.152:0/3004489220 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd180410c0 con 0x7fcd280f5410 2023-12-06T03:53:30.109 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.105+0000 7fcd2de19700 1 -- 172.21.15.152:0/3004489220 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd28045ad0 con 0x7fcd280f5410 2023-12-06T03:53:30.109 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.106+0000 7fcd26ffd700 1 --2- 172.21.15.152:0/3004489220 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd1002ff60 0x7fcd10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:30.110 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.106+0000 7fcd26ffd700 1 --2- 172.21.15.152:0/3004489220 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd1002ff60 0x7fcd10032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fcd1c0096a0 tx=0x7fcd1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:30.111 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.110+0000 7fcd0ffff700 1 -- 172.21.15.152:0/3004489220 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd180095f0 con 0x7fcd280f5410 2023-12-06T03:53:30.312 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.312+0000 7fcd2de19700 1 -- 172.21.15.152:0/3004489220 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd280fae60 con 0x7fcd280f5410 2023-12-06T03:53:30.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.312+0000 7fcd0ffff700 1 -- 172.21.15.152:0/3004489220 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcd1801db70 con 0x7fcd280f5410 2023-12-06T03:53:30.313 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:53:30.313 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:53:30.314 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.314+0000 7fcd2de19700 1 -- 172.21.15.152:0/3004489220 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd1002ff60 msgr2=0x7fcd10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:30.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.314+0000 7fcd2de19700 1 --2- 172.21.15.152:0/3004489220 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd1002ff60 0x7fcd10032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fcd1c0096a0 tx=0x7fcd1c006b40).stop 2023-12-06T03:53:30.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.314+0000 7fcd2de19700 1 -- 172.21.15.152:0/3004489220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd280f5410 msgr2=0x7fcd28105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:30.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.314+0000 7fcd2de19700 1 --2- 172.21.15.152:0/3004489220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd280f5410 0x7fcd28105630 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fcd18013510 tx=0x7fcd1800b040).stop 2023-12-06T03:53:30.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.314+0000 7fcd2de19700 1 -- 172.21.15.152:0/3004489220 shutdown_connections 2023-12-06T03:53:30.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.314+0000 7fcd2de19700 1 --2- 172.21.15.152:0/3004489220 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd1002ff60 0x7fcd10032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:30.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.314+0000 7fcd2de19700 1 --2- 172.21.15.152:0/3004489220 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd280f5410 0x7fcd28105630 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:30.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.314+0000 7fcd2de19700 1 -- 172.21.15.152:0/3004489220 >> 172.21.15.152:0/3004489220 conn(0x7fcd280f0b70 msgr2=0x7fcd280f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:30.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.314+0000 7fcd2de19700 1 -- 172.21.15.152:0/3004489220 shutdown_connections 2023-12-06T03:53:30.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:30.314+0000 7fcd2de19700 1 -- 172.21.15.152:0/3004489220 wait complete. 2023-12-06T03:53:30.315 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:53:30.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:30 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:30.312918+0000 mon.smithi132 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.152:0/3004489220' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:53:31.704 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:53:31.704 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:53:31.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:31 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:30.385396+0000 mgr.smithi132.gokxja (mgr.14162) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:32.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.745+0000 7f2971446700 1 -- 172.21.15.152:0/3964005023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f296c0f5410 msgr2=0x7f296c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:32.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.745+0000 7f2971446700 1 --2- 172.21.15.152:0/3964005023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f296c0f5410 0x7f296c0f5830 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f295c009fb0 tx=0x7f295c009b30).stop 2023-12-06T03:53:32.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.745+0000 7f2971446700 1 -- 172.21.15.152:0/3964005023 shutdown_connections 2023-12-06T03:53:32.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.745+0000 7f2971446700 1 --2- 172.21.15.152:0/3964005023 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f296c0f5410 0x7f296c0f5830 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:32.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.745+0000 7f2971446700 1 -- 172.21.15.152:0/3964005023 >> 172.21.15.152:0/3964005023 conn(0x7f296c0f0b70 msgr2=0x7f296c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:32.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.745+0000 7f2971446700 1 -- 172.21.15.152:0/3964005023 shutdown_connections 2023-12-06T03:53:32.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.745+0000 7f2971446700 1 -- 172.21.15.152:0/3964005023 wait complete. 2023-12-06T03:53:32.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.746+0000 7f2971446700 1 Processor -- start 2023-12-06T03:53:32.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.746+0000 7f2971446700 1 -- start start 2023-12-06T03:53:32.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.746+0000 7f2971446700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f296c0f5410 0x7f296c106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:32.748 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.746+0000 7f2971446700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f296c106bf0 con 0x7f296c0f5410 2023-12-06T03:53:32.749 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.746+0000 7f296affd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f296c0f5410 0x7f296c106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:32.749 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.747+0000 7f296affd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f296c0f5410 0x7f296c106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:57506/0 (socket says 172.21.15.152:57506) 2023-12-06T03:53:32.749 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.747+0000 7f296affd700 1 -- 172.21.15.152:0/2786162141 learned_addr learned my addr 172.21.15.152:0/2786162141 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:53:32.749 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.747+0000 7f296affd700 1 -- 172.21.15.152:0/2786162141 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c005040 con 0x7f296c0f5410 2023-12-06T03:53:32.749 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.747+0000 7f296affd700 1 --2- 172.21.15.152:0/2786162141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f296c0f5410 0x7f296c106700 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f295c013510 tx=0x7f295c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:32.749 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.747+0000 7f29537fe700 1 -- 172.21.15.152:0/2786162141 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f295c009320 con 0x7f296c0f5410 2023-12-06T03:53:32.749 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.747+0000 7f2971446700 1 -- 172.21.15.152:0/2786162141 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f296c106df0 con 0x7f296c0f5410 2023-12-06T03:53:32.750 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.747+0000 7f29537fe700 1 -- 172.21.15.152:0/2786162141 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f295c013f00 con 0x7f296c0f5410 2023-12-06T03:53:32.750 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.747+0000 7f29537fe700 1 -- 172.21.15.152:0/2786162141 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f295c014d90 con 0x7f296c0f5410 2023-12-06T03:53:32.750 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.747+0000 7f2971446700 1 -- 172.21.15.152:0/2786162141 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f296c107280 con 0x7f296c0f5410 2023-12-06T03:53:32.750 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.748+0000 7f29537fe700 1 -- 172.21.15.152:0/2786162141 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f295c0263a0 con 0x7f296c0f5410 2023-12-06T03:53:32.750 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.748+0000 7f29537fe700 1 --2- 172.21.15.152:0/2786162141 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f295402ff60 0x7f2954032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:32.750 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.748+0000 7f29537fe700 1 -- 172.21.15.152:0/2786162141 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f295c0410c0 con 0x7f296c0f5410 2023-12-06T03:53:32.750 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.748+0000 7f2971446700 1 -- 172.21.15.152:0/2786162141 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f296c059100 con 0x7f296c0f5410 2023-12-06T03:53:32.750 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.748+0000 7f296a7fc700 1 --2- 172.21.15.152:0/2786162141 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f295402ff60 0x7f2954032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:32.750 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.749+0000 7f296a7fc700 1 --2- 172.21.15.152:0/2786162141 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f295402ff60 0x7f2954032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f29600096a0 tx=0x7f2960006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:32.753 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.753+0000 7f29537fe700 1 -- 172.21.15.152:0/2786162141 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f295c012920 con 0x7f296c0f5410 2023-12-06T03:53:32.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.958+0000 7f2971446700 1 -- 172.21.15.152:0/2786162141 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f296c0fbf70 con 0x7f296c0f5410 2023-12-06T03:53:32.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.959+0000 7f29537fe700 1 -- 172.21.15.152:0/2786162141 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f295c02b020 con 0x7f296c0f5410 2023-12-06T03:53:32.960 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:53:32.960 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:53:32.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.961+0000 7f2971446700 1 -- 172.21.15.152:0/2786162141 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f295402ff60 msgr2=0x7f2954032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:32.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.961+0000 7f2971446700 1 --2- 172.21.15.152:0/2786162141 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f295402ff60 0x7f2954032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f29600096a0 tx=0x7f2960006b40).stop 2023-12-06T03:53:32.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.961+0000 7f2971446700 1 -- 172.21.15.152:0/2786162141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f296c0f5410 msgr2=0x7f296c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:32.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.961+0000 7f2971446700 1 --2- 172.21.15.152:0/2786162141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f296c0f5410 0x7f296c106700 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f295c013510 tx=0x7f295c00b040).stop 2023-12-06T03:53:32.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.961+0000 7f2971446700 1 -- 172.21.15.152:0/2786162141 shutdown_connections 2023-12-06T03:53:32.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.961+0000 7f2971446700 1 --2- 172.21.15.152:0/2786162141 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f295402ff60 0x7f2954032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:32.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.961+0000 7f2971446700 1 --2- 172.21.15.152:0/2786162141 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f296c0f5410 0x7f296c106700 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:32.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.961+0000 7f2971446700 1 -- 172.21.15.152:0/2786162141 >> 172.21.15.152:0/2786162141 conn(0x7f296c0f0b70 msgr2=0x7f296c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:32.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.961+0000 7f2971446700 1 -- 172.21.15.152:0/2786162141 shutdown_connections 2023-12-06T03:53:32.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:32.961+0000 7f2971446700 1 -- 172.21.15.152:0/2786162141 wait complete. 2023-12-06T03:53:32.963 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:53:33.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:33 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:32.385826+0000 mgr.smithi132.gokxja (mgr.14162) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:33.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:33 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:32.959477+0000 mon.smithi132 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.152:0/2786162141' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:53:34.339 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:53:34.339 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:53:35.456 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.454+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/2987461693 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e180f5410 msgr2=0x7f4e180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:35.456 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.454+0000 7f4e1cbf4700 1 --2- 172.21.15.152:0/2987461693 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e180f5410 0x7f4e180f5830 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f4e0c009fb0 tx=0x7f4e0c009b30).stop 2023-12-06T03:53:35.456 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.454+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/2987461693 shutdown_connections 2023-12-06T03:53:35.456 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.454+0000 7f4e1cbf4700 1 --2- 172.21.15.152:0/2987461693 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e180f5410 0x7f4e180f5830 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:35.456 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.454+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/2987461693 >> 172.21.15.152:0/2987461693 conn(0x7f4e180f0b70 msgr2=0x7f4e180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:35.456 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.455+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/2987461693 shutdown_connections 2023-12-06T03:53:35.456 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.455+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/2987461693 wait complete. 2023-12-06T03:53:35.456 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.455+0000 7f4e1cbf4700 1 Processor -- start 2023-12-06T03:53:35.456 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.455+0000 7f4e1cbf4700 1 -- start start 2023-12-06T03:53:35.456 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.455+0000 7f4e1cbf4700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e180f5410 0x7f4e18109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:35.456 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.455+0000 7f4e1cbf4700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e18060220 con 0x7f4e180f5410 2023-12-06T03:53:35.458 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.455+0000 7f4e1659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e180f5410 0x7f4e18109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:35.458 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.455+0000 7f4e1659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e180f5410 0x7f4e18109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:57524/0 (socket says 172.21.15.152:57524) 2023-12-06T03:53:35.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.455+0000 7f4e1659c700 1 -- 172.21.15.152:0/37195656 learned_addr learned my addr 172.21.15.152:0/37195656 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:53:35.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.456+0000 7f4e1659c700 1 -- 172.21.15.152:0/37195656 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c005040 con 0x7f4e180f5410 2023-12-06T03:53:35.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.456+0000 7f4e1659c700 1 --2- 172.21.15.152:0/37195656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e180f5410 0x7f4e18109420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f4e0c013510 tx=0x7f4e0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:35.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.456+0000 7f4e06ffd700 1 -- 172.21.15.152:0/37195656 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e0c009320 con 0x7f4e180f5410 2023-12-06T03:53:35.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.456+0000 7f4e06ffd700 1 -- 172.21.15.152:0/37195656 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e0c013f00 con 0x7f4e180f5410 2023-12-06T03:53:35.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.456+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/37195656 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e18060420 con 0x7f4e180f5410 2023-12-06T03:53:35.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.456+0000 7f4e06ffd700 1 -- 172.21.15.152:0/37195656 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e0c014d90 con 0x7f4e180f5410 2023-12-06T03:53:35.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.456+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/37195656 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e18109ad0 con 0x7f4e180f5410 2023-12-06T03:53:35.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.457+0000 7f4e06ffd700 1 -- 172.21.15.152:0/37195656 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e0c0095f0 con 0x7f4e180f5410 2023-12-06T03:53:35.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.457+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/37195656 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e18059100 con 0x7f4e180f5410 2023-12-06T03:53:35.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.457+0000 7f4e06ffd700 1 --2- 172.21.15.152:0/37195656 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4e0002ffb0 0x7f4e00032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:35.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.457+0000 7f4e06ffd700 1 -- 172.21.15.152:0/37195656 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e0c040fe0 con 0x7f4e180f5410 2023-12-06T03:53:35.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.457+0000 7f4e15d9b700 1 --2- 172.21.15.152:0/37195656 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4e0002ffb0 0x7f4e00032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:35.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.458+0000 7f4e15d9b700 1 --2- 172.21.15.152:0/37195656 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4e0002ffb0 0x7f4e00032470 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f4e080096a0 tx=0x7f4e08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:35.463 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.462+0000 7f4e06ffd700 1 -- 172.21.15.152:0/37195656 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e0c014420 con 0x7f4e180f5410 2023-12-06T03:53:35.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.670+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/37195656 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e18045ad0 con 0x7f4e180f5410 2023-12-06T03:53:35.672 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.671+0000 7f4e06ffd700 1 -- 172.21.15.152:0/37195656 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4e0c01d7f0 con 0x7f4e180f5410 2023-12-06T03:53:35.672 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:53:35.672 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:53:35.674 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.673+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/37195656 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4e0002ffb0 msgr2=0x7f4e00032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:35.674 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.673+0000 7f4e1cbf4700 1 --2- 172.21.15.152:0/37195656 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4e0002ffb0 0x7f4e00032470 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f4e080096a0 tx=0x7f4e08006b40).stop 2023-12-06T03:53:35.674 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.673+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/37195656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e180f5410 msgr2=0x7f4e18109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:35.674 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.673+0000 7f4e1cbf4700 1 --2- 172.21.15.152:0/37195656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e180f5410 0x7f4e18109420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f4e0c013510 tx=0x7f4e0c00b040).stop 2023-12-06T03:53:35.674 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.673+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/37195656 shutdown_connections 2023-12-06T03:53:35.674 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.673+0000 7f4e1cbf4700 1 --2- 172.21.15.152:0/37195656 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4e0002ffb0 0x7f4e00032470 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:35.674 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.673+0000 7f4e1cbf4700 1 --2- 172.21.15.152:0/37195656 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e180f5410 0x7f4e18109420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:35.674 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.673+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/37195656 >> 172.21.15.152:0/37195656 conn(0x7f4e180f0b70 msgr2=0x7f4e180f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:35.675 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.673+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/37195656 shutdown_connections 2023-12-06T03:53:35.675 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:35.673+0000 7f4e1cbf4700 1 -- 172.21.15.152:0/37195656 wait complete. 2023-12-06T03:53:35.675 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:53:35.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:35 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:34.386186+0000 mgr.smithi132.gokxja (mgr.14162) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:36.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:36 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:35.671630+0000 mon.smithi132 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.152:0/37195656' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:53:37.063 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:53:37.063 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:53:37.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:36.386462+0000 mgr.smithi132.gokxja (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:38.147 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.143+0000 7f59bf59e700 1 -- 172.21.15.152:0/319249992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59c00f62a0 msgr2=0x7f59c00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.143+0000 7f59bf59e700 1 --2- 172.21.15.152:0/319249992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59c00f62a0 0x7f59c00f66c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f59b0009fb0 tx=0x7f59b0009b30).stop 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.143+0000 7f59bf59e700 1 -- 172.21.15.152:0/319249992 shutdown_connections 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.143+0000 7f59bf59e700 1 --2- 172.21.15.152:0/319249992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59c00f62a0 0x7f59c00f66c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.143+0000 7f59bf59e700 1 -- 172.21.15.152:0/319249992 >> 172.21.15.152:0/319249992 conn(0x7f59c00f1aa0 msgr2=0x7f59c00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.143+0000 7f59bf59e700 1 -- 172.21.15.152:0/319249992 shutdown_connections 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.143+0000 7f59bf59e700 1 -- 172.21.15.152:0/319249992 wait complete. 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.143+0000 7f59bf59e700 1 Processor -- start 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.144+0000 7f59bf59e700 1 -- start start 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.144+0000 7f59bf59e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59c00f62a0 0x7f59c01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.144+0000 7f59bf59e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c01069a0 con 0x7f59c00f62a0 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.144+0000 7f59be59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59c00f62a0 0x7f59c01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.144+0000 7f59be59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59c00f62a0 0x7f59c01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:57554/0 (socket says 172.21.15.152:57554) 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.144+0000 7f59be59c700 1 -- 172.21.15.152:0/2048752555 learned_addr learned my addr 172.21.15.152:0/2048752555 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.144+0000 7f59be59c700 1 -- 172.21.15.152:0/2048752555 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59b0005040 con 0x7f59c00f62a0 2023-12-06T03:53:38.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.144+0000 7f59be59c700 1 --2- 172.21.15.152:0/2048752555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59c00f62a0 0x7f59c01064b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f59b0013510 tx=0x7f59b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:38.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.144+0000 7f59aeffd700 1 -- 172.21.15.152:0/2048752555 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59b0009320 con 0x7f59c00f62a0 2023-12-06T03:53:38.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.144+0000 7f59bf59e700 1 -- 172.21.15.152:0/2048752555 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c0106ba0 con 0x7f59c00f62a0 2023-12-06T03:53:38.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.145+0000 7f59aeffd700 1 -- 172.21.15.152:0/2048752555 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f59b0013f00 con 0x7f59c00f62a0 2023-12-06T03:53:38.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.145+0000 7f59aeffd700 1 -- 172.21.15.152:0/2048752555 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59b0014dc0 con 0x7f59c00f62a0 2023-12-06T03:53:38.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.145+0000 7f59bf59e700 1 -- 172.21.15.152:0/2048752555 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c0107030 con 0x7f59c00f62a0 2023-12-06T03:53:38.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.145+0000 7f59aeffd700 1 -- 172.21.15.152:0/2048752555 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59b001d3d0 con 0x7f59c00f62a0 2023-12-06T03:53:38.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.145+0000 7f59aeffd700 1 --2- 172.21.15.152:0/2048752555 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f59a802ff60 0x7f59a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:38.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.145+0000 7f59aeffd700 1 -- 172.21.15.152:0/2048752555 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59b0041400 con 0x7f59c00f62a0 2023-12-06T03:53:38.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.145+0000 7f59bf59e700 1 -- 172.21.15.152:0/2048752555 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c0045ad0 con 0x7f59c00f62a0 2023-12-06T03:53:38.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.146+0000 7f59bdd9b700 1 --2- 172.21.15.152:0/2048752555 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f59a802ff60 0x7f59a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:38.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.146+0000 7f59bdd9b700 1 --2- 172.21.15.152:0/2048752555 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f59a802ff60 0x7f59a8032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f59b40096a0 tx=0x7f59b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:38.151 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.150+0000 7f59aeffd700 1 -- 172.21.15.152:0/2048752555 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59b0012900 con 0x7f59c00f62a0 2023-12-06T03:53:38.360 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.359+0000 7f59bf59e700 1 -- 172.21.15.152:0/2048752555 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59c0107ee0 con 0x7f59c00f62a0 2023-12-06T03:53:38.360 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.360+0000 7f59aeffd700 1 -- 172.21.15.152:0/2048752555 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f59b002b030 con 0x7f59c00f62a0 2023-12-06T03:53:38.361 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:53:38.361 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:53:38.362 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.361+0000 7f59bf59e700 1 -- 172.21.15.152:0/2048752555 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f59a802ff60 msgr2=0x7f59a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:38.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.361+0000 7f59bf59e700 1 --2- 172.21.15.152:0/2048752555 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f59a802ff60 0x7f59a8032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f59b40096a0 tx=0x7f59b4006b40).stop 2023-12-06T03:53:38.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.361+0000 7f59bf59e700 1 -- 172.21.15.152:0/2048752555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59c00f62a0 msgr2=0x7f59c01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:38.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.361+0000 7f59bf59e700 1 --2- 172.21.15.152:0/2048752555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59c00f62a0 0x7f59c01064b0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f59b0013510 tx=0x7f59b000b040).stop 2023-12-06T03:53:38.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.361+0000 7f59bf59e700 1 -- 172.21.15.152:0/2048752555 shutdown_connections 2023-12-06T03:53:38.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.361+0000 7f59bf59e700 1 --2- 172.21.15.152:0/2048752555 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f59a802ff60 0x7f59a8032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:38.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.361+0000 7f59bf59e700 1 --2- 172.21.15.152:0/2048752555 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59c00f62a0 0x7f59c01064b0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:38.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.361+0000 7f59bf59e700 1 -- 172.21.15.152:0/2048752555 >> 172.21.15.152:0/2048752555 conn(0x7f59c00f1aa0 msgr2=0x7f59c00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:38.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.361+0000 7f59bf59e700 1 -- 172.21.15.152:0/2048752555 shutdown_connections 2023-12-06T03:53:38.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:38.361+0000 7f59bf59e700 1 -- 172.21.15.152:0/2048752555 wait complete. 2023-12-06T03:53:38.363 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:53:39.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:38 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:38.360171+0000 mon.smithi132 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.152:0/2048752555' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:53:40.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:39 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:38.386701+0000 mgr.smithi132.gokxja (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:40.116 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:53:40.116 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:53:41.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.169+0000 7fc436b4b700 1 -- 172.21.15.152:0/3044038479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f5410 msgr2=0x7fc4300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:41.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.169+0000 7fc436b4b700 1 --2- 172.21.15.152:0/3044038479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f5410 0x7fc4300f5830 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fc424009fb0 tx=0x7fc424009b30).stop 2023-12-06T03:53:41.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.169+0000 7fc436b4b700 1 -- 172.21.15.152:0/3044038479 shutdown_connections 2023-12-06T03:53:41.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.169+0000 7fc436b4b700 1 --2- 172.21.15.152:0/3044038479 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f5410 0x7fc4300f5830 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:41.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.169+0000 7fc436b4b700 1 -- 172.21.15.152:0/3044038479 >> 172.21.15.152:0/3044038479 conn(0x7fc4300f0b70 msgr2=0x7fc4300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:41.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.169+0000 7fc436b4b700 1 -- 172.21.15.152:0/3044038479 shutdown_connections 2023-12-06T03:53:41.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.169+0000 7fc436b4b700 1 -- 172.21.15.152:0/3044038479 wait complete. 2023-12-06T03:53:41.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.170+0000 7fc436b4b700 1 Processor -- start 2023-12-06T03:53:41.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.170+0000 7fc436b4b700 1 -- start start 2023-12-06T03:53:41.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.170+0000 7fc436b4b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f5410 0x7fc430105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:41.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.170+0000 7fc436b4b700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc430105b20 con 0x7fc4300f5410 2023-12-06T03:53:41.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.170+0000 7fc4348e7700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f5410 0x7fc430105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:41.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.170+0000 7fc4348e7700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f5410 0x7fc430105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:55530/0 (socket says 172.21.15.152:55530) 2023-12-06T03:53:41.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.170+0000 7fc4348e7700 1 -- 172.21.15.152:0/879687979 learned_addr learned my addr 172.21.15.152:0/879687979 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:53:41.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.171+0000 7fc4348e7700 1 -- 172.21.15.152:0/879687979 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc424005040 con 0x7fc4300f5410 2023-12-06T03:53:41.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.171+0000 7fc4348e7700 1 --2- 172.21.15.152:0/879687979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f5410 0x7fc430105630 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fc424013510 tx=0x7fc42400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:41.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.171+0000 7fc42d7fa700 1 -- 172.21.15.152:0/879687979 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc424009320 con 0x7fc4300f5410 2023-12-06T03:53:41.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.171+0000 7fc42d7fa700 1 -- 172.21.15.152:0/879687979 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc424013f00 con 0x7fc4300f5410 2023-12-06T03:53:41.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.171+0000 7fc436b4b700 1 -- 172.21.15.152:0/879687979 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc430105d20 con 0x7fc4300f5410 2023-12-06T03:53:41.175 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.171+0000 7fc42d7fa700 1 -- 172.21.15.152:0/879687979 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc424014d90 con 0x7fc4300f5410 2023-12-06T03:53:41.175 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.171+0000 7fc436b4b700 1 -- 172.21.15.152:0/879687979 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4301061b0 con 0x7fc4300f5410 2023-12-06T03:53:41.175 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.172+0000 7fc42d7fa700 1 -- 172.21.15.152:0/879687979 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc424014420 con 0x7fc4300f5410 2023-12-06T03:53:41.175 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.172+0000 7fc436b4b700 1 -- 172.21.15.152:0/879687979 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc430045ad0 con 0x7fc4300f5410 2023-12-06T03:53:41.175 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.172+0000 7fc42d7fa700 1 --2- 172.21.15.152:0/879687979 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc42002ffb0 0x7fc420032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:41.175 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.172+0000 7fc42d7fa700 1 -- 172.21.15.152:0/879687979 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc424041090 con 0x7fc4300f5410 2023-12-06T03:53:41.175 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.172+0000 7fc42ffff700 1 --2- 172.21.15.152:0/879687979 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc42002ffb0 0x7fc420032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:41.175 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.173+0000 7fc42ffff700 1 --2- 172.21.15.152:0/879687979 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc42002ffb0 0x7fc420032470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fc41c0096a0 tx=0x7fc41c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:41.178 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.177+0000 7fc42d7fa700 1 -- 172.21.15.152:0/879687979 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc42406d2b0 con 0x7fc4300f5410 2023-12-06T03:53:41.387 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.385+0000 7fc436b4b700 1 -- 172.21.15.152:0/879687979 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4300fae60 con 0x7fc4300f5410 2023-12-06T03:53:41.387 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.386+0000 7fc42d7fa700 1 -- 172.21.15.152:0/879687979 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc4240095f0 con 0x7fc4300f5410 2023-12-06T03:53:41.387 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:53:41.388 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:53:41.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.387+0000 7fc436b4b700 1 -- 172.21.15.152:0/879687979 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc42002ffb0 msgr2=0x7fc420032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:41.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.387+0000 7fc436b4b700 1 --2- 172.21.15.152:0/879687979 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc42002ffb0 0x7fc420032470 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fc41c0096a0 tx=0x7fc41c006b40).stop 2023-12-06T03:53:41.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.387+0000 7fc436b4b700 1 -- 172.21.15.152:0/879687979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f5410 msgr2=0x7fc430105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:41.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.387+0000 7fc436b4b700 1 --2- 172.21.15.152:0/879687979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f5410 0x7fc430105630 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fc424013510 tx=0x7fc42400b040).stop 2023-12-06T03:53:41.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.387+0000 7fc436b4b700 1 -- 172.21.15.152:0/879687979 shutdown_connections 2023-12-06T03:53:41.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.387+0000 7fc436b4b700 1 --2- 172.21.15.152:0/879687979 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc42002ffb0 0x7fc420032470 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:41.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.387+0000 7fc436b4b700 1 --2- 172.21.15.152:0/879687979 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f5410 0x7fc430105630 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:41.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.387+0000 7fc436b4b700 1 -- 172.21.15.152:0/879687979 >> 172.21.15.152:0/879687979 conn(0x7fc4300f0b70 msgr2=0x7fc4300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:41.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.388+0000 7fc436b4b700 1 -- 172.21.15.152:0/879687979 shutdown_connections 2023-12-06T03:53:41.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:41.388+0000 7fc436b4b700 1 -- 172.21.15.152:0/879687979 wait complete. 2023-12-06T03:53:41.390 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:53:42.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:41 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:40.386955+0000 mgr.smithi132.gokxja (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:42.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:41 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:41.387128+0000 mon.smithi132 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.152:0/879687979' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:53:42.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:53:42.763 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:53:43.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.891+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/708794025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea80f5410 msgr2=0x7f5ea80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:43.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.891+0000 7f5eaf1ad700 1 --2- 172.21.15.152:0/708794025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea80f5410 0x7f5ea80f5830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f5e9c009fb0 tx=0x7f5e9c009b30).stop 2023-12-06T03:53:43.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.891+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/708794025 shutdown_connections 2023-12-06T03:53:43.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.891+0000 7f5eaf1ad700 1 --2- 172.21.15.152:0/708794025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea80f5410 0x7f5ea80f5830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:43.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.891+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/708794025 >> 172.21.15.152:0/708794025 conn(0x7f5ea80f0b70 msgr2=0x7f5ea80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:43.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.891+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/708794025 shutdown_connections 2023-12-06T03:53:43.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.891+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/708794025 wait complete. 2023-12-06T03:53:43.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.891+0000 7f5eaf1ad700 1 Processor -- start 2023-12-06T03:53:43.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.891+0000 7f5eaf1ad700 1 -- start start 2023-12-06T03:53:43.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.892+0000 7f5eaf1ad700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea80f5410 0x7f5ea8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:43.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.892+0000 7f5eaf1ad700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea8060220 con 0x7f5ea80f5410 2023-12-06T03:53:43.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.892+0000 7f5eacf49700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea80f5410 0x7f5ea8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:43.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.892+0000 7f5eacf49700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea80f5410 0x7f5ea8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:55548/0 (socket says 172.21.15.152:55548) 2023-12-06T03:53:43.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.892+0000 7f5eacf49700 1 -- 172.21.15.152:0/3880868558 learned_addr learned my addr 172.21.15.152:0/3880868558 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:53:43.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.892+0000 7f5eacf49700 1 -- 172.21.15.152:0/3880868558 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e9c005040 con 0x7f5ea80f5410 2023-12-06T03:53:43.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.892+0000 7f5eacf49700 1 --2- 172.21.15.152:0/3880868558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea80f5410 0x7f5ea8109450 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f5e9c013510 tx=0x7f5e9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:43.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.893+0000 7f5ea57fa700 1 -- 172.21.15.152:0/3880868558 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e9c009320 con 0x7f5ea80f5410 2023-12-06T03:53:43.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.893+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/3880868558 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea8060420 con 0x7f5ea80f5410 2023-12-06T03:53:43.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.893+0000 7f5ea57fa700 1 -- 172.21.15.152:0/3880868558 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e9c013f00 con 0x7f5ea80f5410 2023-12-06T03:53:43.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.893+0000 7f5ea57fa700 1 -- 172.21.15.152:0/3880868558 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e9c014d90 con 0x7f5ea80f5410 2023-12-06T03:53:43.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.893+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/3880868558 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea8109b00 con 0x7f5ea80f5410 2023-12-06T03:53:43.898 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.893+0000 7f5ea57fa700 1 -- 172.21.15.152:0/3880868558 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e9c0263a0 con 0x7f5ea80f5410 2023-12-06T03:53:43.898 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.894+0000 7f5ea57fa700 1 --2- 172.21.15.152:0/3880868558 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5e9802ff60 0x7f5e98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:43.898 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.894+0000 7f5ea57fa700 1 -- 172.21.15.152:0/3880868558 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e9c0410e0 con 0x7f5ea80f5410 2023-12-06T03:53:43.898 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.894+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/3880868558 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea8059100 con 0x7f5ea80f5410 2023-12-06T03:53:43.898 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.894+0000 7f5ea7fff700 1 --2- 172.21.15.152:0/3880868558 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5e9802ff60 0x7f5e98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:43.898 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.894+0000 7f5ea7fff700 1 --2- 172.21.15.152:0/3880868558 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5e9802ff60 0x7f5e98032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5e940096a0 tx=0x7f5e94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:43.900 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:43.899+0000 7f5ea57fa700 1 -- 172.21.15.152:0/3880868558 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e9c012920 con 0x7f5ea80f5410 2023-12-06T03:53:44.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:42.387312+0000 mgr.smithi132.gokxja (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:44.109 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:44.107+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/3880868558 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ea8045ad0 con 0x7f5ea80f5410 2023-12-06T03:53:44.110 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:44.108+0000 7f5ea57fa700 1 -- 172.21.15.152:0/3880868558 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5e9c02b020 con 0x7f5ea80f5410 2023-12-06T03:53:44.110 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:53:44.110 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:53:44.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:44.110+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/3880868558 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5e9802ff60 msgr2=0x7f5e98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:44.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:44.110+0000 7f5eaf1ad700 1 --2- 172.21.15.152:0/3880868558 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5e9802ff60 0x7f5e98032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f5e940096a0 tx=0x7f5e94006b40).stop 2023-12-06T03:53:44.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:44.110+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/3880868558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea80f5410 msgr2=0x7f5ea8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:44.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:44.110+0000 7f5eaf1ad700 1 --2- 172.21.15.152:0/3880868558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea80f5410 0x7f5ea8109450 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f5e9c013510 tx=0x7f5e9c00b040).stop 2023-12-06T03:53:44.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:44.110+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/3880868558 shutdown_connections 2023-12-06T03:53:44.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:44.110+0000 7f5eaf1ad700 1 --2- 172.21.15.152:0/3880868558 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5e9802ff60 0x7f5e98032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:44.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:44.110+0000 7f5eaf1ad700 1 --2- 172.21.15.152:0/3880868558 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5ea80f5410 0x7f5ea8109450 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:44.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:44.110+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/3880868558 >> 172.21.15.152:0/3880868558 conn(0x7f5ea80f0b70 msgr2=0x7f5ea80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:44.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:44.110+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/3880868558 shutdown_connections 2023-12-06T03:53:44.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:44.110+0000 7f5eaf1ad700 1 -- 172.21.15.152:0/3880868558 wait complete. 2023-12-06T03:53:44.113 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:53:45.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:44 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:44.109531+0000 mon.smithi132 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.152:0/3880868558' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:53:46.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:45 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:44.387618+0000 mgr.smithi132.gokxja (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:46.618 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:53:46.618 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:53:48.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:47 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:46.387912+0000 mgr.smithi132.gokxja (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:48.861 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.859+0000 7f37ca72d700 1 -- 172.21.15.152:0/1376265788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37c40f6510 msgr2=0x7f37c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:48.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.859+0000 7f37ca72d700 1 --2- 172.21.15.152:0/1376265788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37c40f6510 0x7f37c40f6930 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f37b4009960 tx=0x7f37b4004d10).stop 2023-12-06T03:53:48.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.859+0000 7f37ca72d700 1 -- 172.21.15.152:0/1376265788 shutdown_connections 2023-12-06T03:53:48.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.859+0000 7f37ca72d700 1 --2- 172.21.15.152:0/1376265788 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37c40f6510 0x7f37c40f6930 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:48.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.859+0000 7f37ca72d700 1 -- 172.21.15.152:0/1376265788 >> 172.21.15.152:0/1376265788 conn(0x7f37c40f1c70 msgr2=0x7f37c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:48.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.859+0000 7f37ca72d700 1 -- 172.21.15.152:0/1376265788 shutdown_connections 2023-12-06T03:53:48.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.859+0000 7f37ca72d700 1 -- 172.21.15.152:0/1376265788 wait complete. 2023-12-06T03:53:48.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.860+0000 7f37ca72d700 1 Processor -- start 2023-12-06T03:53:48.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.860+0000 7f37ca72d700 1 -- start start 2023-12-06T03:53:48.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.860+0000 7f37ca72d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37c40f6510 0x7f37c4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:48.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.860+0000 7f37ca72d700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37c4106c20 con 0x7f37c40f6510 2023-12-06T03:53:48.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.860+0000 7f37c3fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37c40f6510 0x7f37c4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:48.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.860+0000 7f37c3fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37c40f6510 0x7f37c4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:55566/0 (socket says 172.21.15.152:55566) 2023-12-06T03:53:48.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.860+0000 7f37c3fff700 1 -- 172.21.15.152:0/1886573059 learned_addr learned my addr 172.21.15.152:0/1886573059 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:53:48.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.861+0000 7f37c3fff700 1 -- 172.21.15.152:0/1886573059 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37b4009640 con 0x7f37c40f6510 2023-12-06T03:53:48.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.861+0000 7f37c3fff700 1 --2- 172.21.15.152:0/1886573059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37c40f6510 0x7f37c4106730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f37b4004b70 tx=0x7f37b4004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:48.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.861+0000 7f37c17fa700 1 -- 172.21.15.152:0/1886573059 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37b4018070 con 0x7f37c40f6510 2023-12-06T03:53:48.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.861+0000 7f37c17fa700 1 -- 172.21.15.152:0/1886573059 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37b400b870 con 0x7f37c40f6510 2023-12-06T03:53:48.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.861+0000 7f37ca72d700 1 -- 172.21.15.152:0/1886573059 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37c4106e20 con 0x7f37c40f6510 2023-12-06T03:53:48.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.861+0000 7f37c17fa700 1 -- 172.21.15.152:0/1886573059 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37b401c560 con 0x7f37c40f6510 2023-12-06T03:53:48.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.861+0000 7f37ca72d700 1 -- 172.21.15.152:0/1886573059 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37c41072b0 con 0x7f37c40f6510 2023-12-06T03:53:48.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.862+0000 7f37c17fa700 1 -- 172.21.15.152:0/1886573059 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37b401c6c0 con 0x7f37c40f6510 2023-12-06T03:53:48.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.862+0000 7f37c17fa700 1 --2- 172.21.15.152:0/1886573059 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f37b002ff60 0x7f37b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:48.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.862+0000 7f37c17fa700 1 -- 172.21.15.152:0/1886573059 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37b4041340 con 0x7f37c40f6510 2023-12-06T03:53:48.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.862+0000 7f37ca72d700 1 -- 172.21.15.152:0/1886573059 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37c4045ad0 con 0x7f37c40f6510 2023-12-06T03:53:48.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.862+0000 7f37bb7fe700 1 --2- 172.21.15.152:0/1886573059 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f37b002ff60 0x7f37b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:48.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.863+0000 7f37bb7fe700 1 --2- 172.21.15.152:0/1886573059 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f37b002ff60 0x7f37b0032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f37ac0096a0 tx=0x7f37ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:48.869 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:48.867+0000 7f37c17fa700 1 -- 172.21.15.152:0/1886573059 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37b4012c00 con 0x7f37c40f6510 2023-12-06T03:53:49.077 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:49.075+0000 7f37ca72d700 1 -- 172.21.15.152:0/1886573059 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37c40fc050 con 0x7f37c40f6510 2023-12-06T03:53:49.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:49.076+0000 7f37c17fa700 1 -- 172.21.15.152:0/1886573059 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f37b40127f0 con 0x7f37c40f6510 2023-12-06T03:53:49.078 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:53:49.078 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:53:49.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:49.078+0000 7f37ca72d700 1 -- 172.21.15.152:0/1886573059 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f37b002ff60 msgr2=0x7f37b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:49.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:49.078+0000 7f37ca72d700 1 --2- 172.21.15.152:0/1886573059 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f37b002ff60 0x7f37b0032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f37ac0096a0 tx=0x7f37ac006b40).stop 2023-12-06T03:53:49.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:49.078+0000 7f37ca72d700 1 -- 172.21.15.152:0/1886573059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37c40f6510 msgr2=0x7f37c4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:49.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:49.078+0000 7f37ca72d700 1 --2- 172.21.15.152:0/1886573059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37c40f6510 0x7f37c4106730 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f37b4004b70 tx=0x7f37b4004d10).stop 2023-12-06T03:53:49.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:49.078+0000 7f37ca72d700 1 -- 172.21.15.152:0/1886573059 shutdown_connections 2023-12-06T03:53:49.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:49.078+0000 7f37ca72d700 1 --2- 172.21.15.152:0/1886573059 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f37b002ff60 0x7f37b0032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:49.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:49.078+0000 7f37ca72d700 1 --2- 172.21.15.152:0/1886573059 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37c40f6510 0x7f37c4106730 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:49.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:49.078+0000 7f37ca72d700 1 -- 172.21.15.152:0/1886573059 >> 172.21.15.152:0/1886573059 conn(0x7f37c40f1c70 msgr2=0x7f37c40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:49.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:49.078+0000 7f37ca72d700 1 -- 172.21.15.152:0/1886573059 shutdown_connections 2023-12-06T03:53:49.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:49.078+0000 7f37ca72d700 1 -- 172.21.15.152:0/1886573059 wait complete. 2023-12-06T03:53:49.081 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:53:50.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:49 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:48.388224+0000 mgr.smithi132.gokxja (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:50.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:49 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:49.077674+0000 mon.smithi132 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.152:0/1886573059' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:53:50.449 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:53:50.450 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:53:51.541 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.537+0000 7f071da5e700 1 -- 172.21.15.152:0/3533258868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07180f6480 msgr2=0x7f07180f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:51.541 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.537+0000 7f071da5e700 1 --2- 172.21.15.152:0/3533258868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07180f6480 0x7f07180f68a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f070c009fb0 tx=0x7f070c009b30).stop 2023-12-06T03:53:51.541 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.537+0000 7f071da5e700 1 -- 172.21.15.152:0/3533258868 shutdown_connections 2023-12-06T03:53:51.541 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.537+0000 7f071da5e700 1 --2- 172.21.15.152:0/3533258868 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07180f6480 0x7f07180f68a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:51.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.537+0000 7f071da5e700 1 -- 172.21.15.152:0/3533258868 >> 172.21.15.152:0/3533258868 conn(0x7f07180f1c00 msgr2=0x7f07180f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:51.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.537+0000 7f071da5e700 1 -- 172.21.15.152:0/3533258868 shutdown_connections 2023-12-06T03:53:51.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.537+0000 7f071da5e700 1 -- 172.21.15.152:0/3533258868 wait complete. 2023-12-06T03:53:51.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.538+0000 7f071da5e700 1 Processor -- start 2023-12-06T03:53:51.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.538+0000 7f071da5e700 1 -- start start 2023-12-06T03:53:51.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.538+0000 7f071da5e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07180f6480 0x7f07181066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:51.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.538+0000 7f071da5e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0718106bc0 con 0x7f07180f6480 2023-12-06T03:53:51.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.538+0000 7f0716ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07180f6480 0x7f07181066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:51.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.538+0000 7f0716ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07180f6480 0x7f07181066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:58776/0 (socket says 172.21.15.152:58776) 2023-12-06T03:53:51.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.538+0000 7f0716ffd700 1 -- 172.21.15.152:0/663048041 learned_addr learned my addr 172.21.15.152:0/663048041 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:53:51.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.539+0000 7f0716ffd700 1 -- 172.21.15.152:0/663048041 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f070c005040 con 0x7f07180f6480 2023-12-06T03:53:51.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.539+0000 7f0716ffd700 1 --2- 172.21.15.152:0/663048041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07180f6480 0x7f07181066d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f070c013510 tx=0x7f070c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:51.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.539+0000 7f06ff7fe700 1 -- 172.21.15.152:0/663048041 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f070c009320 con 0x7f07180f6480 2023-12-06T03:53:51.543 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.539+0000 7f071da5e700 1 -- 172.21.15.152:0/663048041 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0718106dc0 con 0x7f07180f6480 2023-12-06T03:53:51.543 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.539+0000 7f06ff7fe700 1 -- 172.21.15.152:0/663048041 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f070c013f00 con 0x7f07180f6480 2023-12-06T03:53:51.543 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.539+0000 7f06ff7fe700 1 -- 172.21.15.152:0/663048041 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f070c014d90 con 0x7f07180f6480 2023-12-06T03:53:51.543 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.539+0000 7f071da5e700 1 -- 172.21.15.152:0/663048041 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0718107250 con 0x7f07180f6480 2023-12-06T03:53:51.544 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.540+0000 7f06ff7fe700 1 -- 172.21.15.152:0/663048041 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f070c014420 con 0x7f07180f6480 2023-12-06T03:53:51.544 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.540+0000 7f071da5e700 1 -- 172.21.15.152:0/663048041 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0718045ad0 con 0x7f07180f6480 2023-12-06T03:53:51.544 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.540+0000 7f06ff7fe700 1 --2- 172.21.15.152:0/663048041 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f070002ff60 0x7f0700032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:51.544 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.540+0000 7f06ff7fe700 1 -- 172.21.15.152:0/663048041 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f070c0410c0 con 0x7f07180f6480 2023-12-06T03:53:51.544 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.540+0000 7f07167fc700 1 --2- 172.21.15.152:0/663048041 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f070002ff60 0x7f0700032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:51.544 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.541+0000 7f07167fc700 1 --2- 172.21.15.152:0/663048041 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f070002ff60 0x7f0700032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f07080096a0 tx=0x7f0708006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:51.546 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.545+0000 7f06ff7fe700 1 -- 172.21.15.152:0/663048041 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f070c0095f0 con 0x7f07180f6480 2023-12-06T03:53:51.750 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.748+0000 7f071da5e700 1 -- 172.21.15.152:0/663048041 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0718108100 con 0x7f07180f6480 2023-12-06T03:53:51.750 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.749+0000 7f06ff7fe700 1 -- 172.21.15.152:0/663048041 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f070c01db70 con 0x7f07180f6480 2023-12-06T03:53:51.751 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:53:51.751 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:53:51.752 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.750+0000 7f071da5e700 1 -- 172.21.15.152:0/663048041 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f070002ff60 msgr2=0x7f0700032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:51.752 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.750+0000 7f071da5e700 1 --2- 172.21.15.152:0/663048041 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f070002ff60 0x7f0700032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f07080096a0 tx=0x7f0708006b40).stop 2023-12-06T03:53:51.752 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.750+0000 7f071da5e700 1 -- 172.21.15.152:0/663048041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07180f6480 msgr2=0x7f07181066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:51.752 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.750+0000 7f071da5e700 1 --2- 172.21.15.152:0/663048041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07180f6480 0x7f07181066d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f070c013510 tx=0x7f070c00b040).stop 2023-12-06T03:53:51.752 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.750+0000 7f071da5e700 1 -- 172.21.15.152:0/663048041 shutdown_connections 2023-12-06T03:53:51.752 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.750+0000 7f071da5e700 1 --2- 172.21.15.152:0/663048041 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f070002ff60 0x7f0700032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:51.752 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.750+0000 7f071da5e700 1 --2- 172.21.15.152:0/663048041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f07180f6480 0x7f07181066d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:51.753 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.750+0000 7f071da5e700 1 -- 172.21.15.152:0/663048041 >> 172.21.15.152:0/663048041 conn(0x7f07180f1c00 msgr2=0x7f07180f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:51.753 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.750+0000 7f071da5e700 1 -- 172.21.15.152:0/663048041 shutdown_connections 2023-12-06T03:53:51.753 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:51.750+0000 7f071da5e700 1 -- 172.21.15.152:0/663048041 wait complete. 2023-12-06T03:53:51.753 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:53:52.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:51 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:50.388495+0000 mgr.smithi132.gokxja (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:53.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:52 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:51.750237+0000 mon.smithi132 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.152:0/663048041' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:53:53.123 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:53:53.123 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:53:54.110 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:53 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:52.388843+0000 mgr.smithi132.gokxja (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:55.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:54.516241+0000 mon.smithi132 (mon.0) 129 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/mirror_snapshot_schedule"}]: dispatch 2023-12-06T03:53:55.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:54.525263+0000 mon.smithi132 (mon.0) 130 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/trash_purge_schedule"}]: dispatch 2023-12-06T03:53:56.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:54.389170+0000 mgr.smithi132.gokxja (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:58.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:56.389515+0000 mgr.smithi132.gokxja (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:53:58.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.983+0000 7f3edd0ec700 1 -- 172.21.15.152:0/2868057823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed80f5410 msgr2=0x7f3ed80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:58.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.983+0000 7f3edd0ec700 1 --2- 172.21.15.152:0/2868057823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed80f5410 0x7f3ed80f5830 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f3ec0009fb0 tx=0x7f3ec0009b30).stop 2023-12-06T03:53:58.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.983+0000 7f3edd0ec700 1 -- 172.21.15.152:0/2868057823 shutdown_connections 2023-12-06T03:53:58.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.983+0000 7f3edd0ec700 1 --2- 172.21.15.152:0/2868057823 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed80f5410 0x7f3ed80f5830 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:58.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.983+0000 7f3edd0ec700 1 -- 172.21.15.152:0/2868057823 >> 172.21.15.152:0/2868057823 conn(0x7f3ed80f0b70 msgr2=0x7f3ed80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:58.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.983+0000 7f3edd0ec700 1 -- 172.21.15.152:0/2868057823 shutdown_connections 2023-12-06T03:53:58.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.983+0000 7f3edd0ec700 1 -- 172.21.15.152:0/2868057823 wait complete. 2023-12-06T03:53:58.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.984+0000 7f3edd0ec700 1 Processor -- start 2023-12-06T03:53:58.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.984+0000 7f3edd0ec700 1 -- start start 2023-12-06T03:53:58.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.984+0000 7f3edd0ec700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed80f5410 0x7f3ed8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:58.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.984+0000 7f3edd0ec700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed8060240 con 0x7f3ed80f5410 2023-12-06T03:53:58.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.984+0000 7f3ed6d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed80f5410 0x7f3ed8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:58.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.985+0000 7f3ed6d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed80f5410 0x7f3ed8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:58790/0 (socket says 172.21.15.152:58790) 2023-12-06T03:53:58.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.985+0000 7f3ed6d9d700 1 -- 172.21.15.152:0/857875245 learned_addr learned my addr 172.21.15.152:0/857875245 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:53:58.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.985+0000 7f3ed6d9d700 1 -- 172.21.15.152:0/857875245 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec0005040 con 0x7f3ed80f5410 2023-12-06T03:53:58.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.985+0000 7f3ed6d9d700 1 --2- 172.21.15.152:0/857875245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed80f5410 0x7f3ed8109430 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f3ec0014000 tx=0x7f3ec000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:58.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.985+0000 7f3ecf7fe700 1 -- 172.21.15.152:0/857875245 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ec00095f0 con 0x7f3ed80f5410 2023-12-06T03:53:58.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.985+0000 7f3edd0ec700 1 -- 172.21.15.152:0/857875245 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed8060440 con 0x7f3ed80f5410 2023-12-06T03:53:58.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.985+0000 7f3ecf7fe700 1 -- 172.21.15.152:0/857875245 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ec0007e70 con 0x7f3ed80f5410 2023-12-06T03:53:58.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.985+0000 7f3ecf7fe700 1 -- 172.21.15.152:0/857875245 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ec0014ea0 con 0x7f3ed80f5410 2023-12-06T03:53:58.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.985+0000 7f3edd0ec700 1 -- 172.21.15.152:0/857875245 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed8109ae0 con 0x7f3ed80f5410 2023-12-06T03:53:58.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.986+0000 7f3ecf7fe700 1 -- 172.21.15.152:0/857875245 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ec0025070 con 0x7f3ed80f5410 2023-12-06T03:53:58.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.986+0000 7f3edd0ec700 1 -- 172.21.15.152:0/857875245 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed8045ad0 con 0x7f3ed80f5410 2023-12-06T03:53:58.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.986+0000 7f3ecf7fe700 1 --2- 172.21.15.152:0/857875245 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ec402ff60 0x7f3ec4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:53:58.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.986+0000 7f3ecf7fe700 1 -- 172.21.15.152:0/857875245 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ec0040a30 con 0x7f3ed80f5410 2023-12-06T03:53:58.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.986+0000 7f3ed659c700 1 --2- 172.21.15.152:0/857875245 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ec402ff60 0x7f3ec4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:53:58.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.987+0000 7f3ed659c700 1 --2- 172.21.15.152:0/857875245 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ec402ff60 0x7f3ec4032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f3ec80096a0 tx=0x7f3ec8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:53:58.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:57.991+0000 7f3ecf7fe700 1 -- 172.21.15.152:0/857875245 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ec00079c0 con 0x7f3ed80f5410 2023-12-06T03:53:58.202 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:58.198+0000 7f3edd0ec700 1 -- 172.21.15.152:0/857875245 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ed80fae60 con 0x7f3ed80f5410 2023-12-06T03:53:58.202 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:58.199+0000 7f3ecf7fe700 1 -- 172.21.15.152:0/857875245 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ec000f300 con 0x7f3ed80f5410 2023-12-06T03:53:58.202 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:53:58.203 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:53:58.203 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:58.201+0000 7f3edd0ec700 1 -- 172.21.15.152:0/857875245 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ec402ff60 msgr2=0x7f3ec4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:58.203 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:58.201+0000 7f3edd0ec700 1 --2- 172.21.15.152:0/857875245 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ec402ff60 0x7f3ec4032420 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f3ec80096a0 tx=0x7f3ec8006b40).stop 2023-12-06T03:53:58.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:58.201+0000 7f3edd0ec700 1 -- 172.21.15.152:0/857875245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed80f5410 msgr2=0x7f3ed8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:53:58.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:58.201+0000 7f3edd0ec700 1 --2- 172.21.15.152:0/857875245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed80f5410 0x7f3ed8109430 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f3ec0014000 tx=0x7f3ec000b040).stop 2023-12-06T03:53:58.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:58.201+0000 7f3edd0ec700 1 -- 172.21.15.152:0/857875245 shutdown_connections 2023-12-06T03:53:58.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:58.201+0000 7f3edd0ec700 1 --2- 172.21.15.152:0/857875245 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ec402ff60 0x7f3ec4032420 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:58.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:58.201+0000 7f3edd0ec700 1 --2- 172.21.15.152:0/857875245 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3ed80f5410 0x7f3ed8109430 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:53:58.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:58.201+0000 7f3edd0ec700 1 -- 172.21.15.152:0/857875245 >> 172.21.15.152:0/857875245 conn(0x7f3ed80f0b70 msgr2=0x7f3ed80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:53:58.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:58.201+0000 7f3edd0ec700 1 -- 172.21.15.152:0/857875245 shutdown_connections 2023-12-06T03:53:58.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:53:58.201+0000 7f3edd0ec700 1 -- 172.21.15.152:0/857875245 wait complete. 2023-12-06T03:53:58.204 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:53:59.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:58 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:53:58.200543+0000 mon.smithi132 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.152:0/857875245' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:00.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:53:59 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:53:58.389852+0000 mgr.smithi132.gokxja (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:00.590 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:00.590 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:01.717 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.714+0000 7f0b23cea700 1 -- 172.21.15.152:0/1341997397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b1c0f5410 msgr2=0x7f0b1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:01.717 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.714+0000 7f0b23cea700 1 --2- 172.21.15.152:0/1341997397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b1c0f5410 0x7f0b1c0f5830 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f0b18009fb0 tx=0x7f0b18009b30).stop 2023-12-06T03:54:01.717 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.714+0000 7f0b23cea700 1 -- 172.21.15.152:0/1341997397 shutdown_connections 2023-12-06T03:54:01.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.714+0000 7f0b23cea700 1 --2- 172.21.15.152:0/1341997397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b1c0f5410 0x7f0b1c0f5830 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:01.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.714+0000 7f0b23cea700 1 -- 172.21.15.152:0/1341997397 >> 172.21.15.152:0/1341997397 conn(0x7f0b1c0f0b70 msgr2=0x7f0b1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:01.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.714+0000 7f0b23cea700 1 -- 172.21.15.152:0/1341997397 shutdown_connections 2023-12-06T03:54:01.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.714+0000 7f0b23cea700 1 -- 172.21.15.152:0/1341997397 wait complete. 2023-12-06T03:54:01.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.715+0000 7f0b23cea700 1 Processor -- start 2023-12-06T03:54:01.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.715+0000 7f0b23cea700 1 -- start start 2023-12-06T03:54:01.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.715+0000 7f0b23cea700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b1c0f5410 0x7f0b1c0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:01.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.715+0000 7f0b23cea700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b1c0fb820 con 0x7f0b1c0f5410 2023-12-06T03:54:01.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.715+0000 7f0b21a86700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b1c0f5410 0x7f0b1c0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:01.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.715+0000 7f0b21a86700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b1c0f5410 0x7f0b1c0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:35712/0 (socket says 172.21.15.152:35712) 2023-12-06T03:54:01.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.715+0000 7f0b21a86700 1 -- 172.21.15.152:0/2075771466 learned_addr learned my addr 172.21.15.152:0/2075771466 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:01.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.715+0000 7f0b21a86700 1 -- 172.21.15.152:0/2075771466 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b18005040 con 0x7f0b1c0f5410 2023-12-06T03:54:01.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.715+0000 7f0b21a86700 1 --2- 172.21.15.152:0/2075771466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b1c0f5410 0x7f0b1c0fd3a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0b18013510 tx=0x7f0b1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:01.719 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.715+0000 7f0b0e7fc700 1 -- 172.21.15.152:0/2075771466 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b18009320 con 0x7f0b1c0f5410 2023-12-06T03:54:01.719 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.716+0000 7f0b23cea700 1 -- 172.21.15.152:0/2075771466 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b1c0fb9c0 con 0x7f0b1c0f5410 2023-12-06T03:54:01.719 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.716+0000 7f0b0e7fc700 1 -- 172.21.15.152:0/2075771466 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0b18013f00 con 0x7f0b1c0f5410 2023-12-06T03:54:01.719 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.716+0000 7f0b0e7fc700 1 -- 172.21.15.152:0/2075771466 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b18014d90 con 0x7f0b1c0f5410 2023-12-06T03:54:01.719 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.716+0000 7f0b23cea700 1 -- 172.21.15.152:0/2075771466 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b1c0fbe50 con 0x7f0b1c0f5410 2023-12-06T03:54:01.720 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.716+0000 7f0b0e7fc700 1 -- 172.21.15.152:0/2075771466 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0b180263a0 con 0x7f0b1c0f5410 2023-12-06T03:54:01.720 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.716+0000 7f0b0e7fc700 1 --2- 172.21.15.152:0/2075771466 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f0b0802ff60 0x7f0b08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:01.720 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.717+0000 7f0b0e7fc700 1 -- 172.21.15.152:0/2075771466 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0b180410e0 con 0x7f0b1c0f5410 2023-12-06T03:54:01.720 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.717+0000 7f0b23cea700 1 -- 172.21.15.152:0/2075771466 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b1c059100 con 0x7f0b1c0f5410 2023-12-06T03:54:01.720 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.717+0000 7f0b21285700 1 --2- 172.21.15.152:0/2075771466 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f0b0802ff60 0x7f0b08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:01.720 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.717+0000 7f0b21285700 1 --2- 172.21.15.152:0/2075771466 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f0b0802ff60 0x7f0b08032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f0b100096a0 tx=0x7f0b10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:01.723 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.721+0000 7f0b0e7fc700 1 -- 172.21.15.152:0/2075771466 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0b18012920 con 0x7f0b1c0f5410 2023-12-06T03:54:01.931 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.929+0000 7f0b23cea700 1 -- 172.21.15.152:0/2075771466 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0b1c0fc550 con 0x7f0b1c0f5410 2023-12-06T03:54:01.931 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.930+0000 7f0b0e7fc700 1 -- 172.21.15.152:0/2075771466 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0b1802b020 con 0x7f0b1c0f5410 2023-12-06T03:54:01.932 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:01.932 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:01.933 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.932+0000 7f0b23cea700 1 -- 172.21.15.152:0/2075771466 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f0b0802ff60 msgr2=0x7f0b08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:01.933 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.932+0000 7f0b23cea700 1 --2- 172.21.15.152:0/2075771466 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f0b0802ff60 0x7f0b08032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f0b100096a0 tx=0x7f0b10006b40).stop 2023-12-06T03:54:01.933 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.932+0000 7f0b23cea700 1 -- 172.21.15.152:0/2075771466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b1c0f5410 msgr2=0x7f0b1c0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:01.933 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.932+0000 7f0b23cea700 1 --2- 172.21.15.152:0/2075771466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b1c0f5410 0x7f0b1c0fd3a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0b18013510 tx=0x7f0b1800b040).stop 2023-12-06T03:54:01.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.932+0000 7f0b23cea700 1 -- 172.21.15.152:0/2075771466 shutdown_connections 2023-12-06T03:54:01.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.932+0000 7f0b23cea700 1 --2- 172.21.15.152:0/2075771466 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f0b0802ff60 0x7f0b08032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:01.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.932+0000 7f0b23cea700 1 --2- 172.21.15.152:0/2075771466 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0b1c0f5410 0x7f0b1c0fd3a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:01.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.932+0000 7f0b23cea700 1 -- 172.21.15.152:0/2075771466 >> 172.21.15.152:0/2075771466 conn(0x7f0b1c0f0b70 msgr2=0x7f0b1c05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:01.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.932+0000 7f0b23cea700 1 -- 172.21.15.152:0/2075771466 shutdown_connections 2023-12-06T03:54:01.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:01.932+0000 7f0b23cea700 1 -- 172.21.15.152:0/2075771466 wait complete. 2023-12-06T03:54:01.934 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:02.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:01 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:00.390129+0000 mgr.smithi132.gokxja (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:03.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:01.931284+0000 mon.smithi132 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.152:0/2075771466' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:03.312 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:03.312 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:04.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:02.390477+0000 mgr.smithi132.gokxja (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:04.443 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.441+0000 7f90c6492700 1 -- 172.21.15.152:0/2356977283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90c00f5410 msgr2=0x7f90c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:04.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.441+0000 7f90c6492700 1 --2- 172.21.15.152:0/2356977283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90c00f5410 0x7f90c00f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f90a8009fb0 tx=0x7f90a8009b30).stop 2023-12-06T03:54:04.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.441+0000 7f90c6492700 1 -- 172.21.15.152:0/2356977283 shutdown_connections 2023-12-06T03:54:04.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.441+0000 7f90c6492700 1 --2- 172.21.15.152:0/2356977283 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90c00f5410 0x7f90c00f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:04.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.441+0000 7f90c6492700 1 -- 172.21.15.152:0/2356977283 >> 172.21.15.152:0/2356977283 conn(0x7f90c00f0b70 msgr2=0x7f90c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:04.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.441+0000 7f90c6492700 1 -- 172.21.15.152:0/2356977283 shutdown_connections 2023-12-06T03:54:04.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.441+0000 7f90c6492700 1 -- 172.21.15.152:0/2356977283 wait complete. 2023-12-06T03:54:04.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.442+0000 7f90c6492700 1 Processor -- start 2023-12-06T03:54:04.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.442+0000 7f90c6492700 1 -- start start 2023-12-06T03:54:04.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.442+0000 7f90c6492700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90c00f5410 0x7f90c0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:04.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.442+0000 7f90c6492700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90c0060220 con 0x7f90c00f5410 2023-12-06T03:54:04.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.442+0000 7f90bffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90c00f5410 0x7f90c0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:04.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.442+0000 7f90bffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90c00f5410 0x7f90c0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:35722/0 (socket says 172.21.15.152:35722) 2023-12-06T03:54:04.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.442+0000 7f90bffff700 1 -- 172.21.15.152:0/2229843420 learned_addr learned my addr 172.21.15.152:0/2229843420 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:04.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.442+0000 7f90bffff700 1 -- 172.21.15.152:0/2229843420 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90a8005040 con 0x7f90c00f5410 2023-12-06T03:54:04.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.443+0000 7f90bffff700 1 --2- 172.21.15.152:0/2229843420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90c00f5410 0x7f90c0109450 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f90a8014000 tx=0x7f90a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:04.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.443+0000 7f90bcff9700 1 -- 172.21.15.152:0/2229843420 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90a80095f0 con 0x7f90c00f5410 2023-12-06T03:54:04.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.443+0000 7f90bcff9700 1 -- 172.21.15.152:0/2229843420 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90a8007e70 con 0x7f90c00f5410 2023-12-06T03:54:04.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.443+0000 7f90c6492700 1 -- 172.21.15.152:0/2229843420 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90c0060420 con 0x7f90c00f5410 2023-12-06T03:54:04.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.443+0000 7f90bcff9700 1 -- 172.21.15.152:0/2229843420 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90a8014ea0 con 0x7f90c00f5410 2023-12-06T03:54:04.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.443+0000 7f90c6492700 1 -- 172.21.15.152:0/2229843420 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90c0109b00 con 0x7f90c00f5410 2023-12-06T03:54:04.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.444+0000 7f90bcff9700 1 -- 172.21.15.152:0/2229843420 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f90a8025070 con 0x7f90c00f5410 2023-12-06T03:54:04.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.444+0000 7f90c6492700 1 -- 172.21.15.152:0/2229843420 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90c0059100 con 0x7f90c00f5410 2023-12-06T03:54:04.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.444+0000 7f90bcff9700 1 --2- 172.21.15.152:0/2229843420 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f90ac02ff60 0x7f90ac032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:04.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.444+0000 7f90bcff9700 1 -- 172.21.15.152:0/2229843420 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f90a8040a20 con 0x7f90c00f5410 2023-12-06T03:54:04.447 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.444+0000 7f90bf7fe700 1 --2- 172.21.15.152:0/2229843420 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f90ac02ff60 0x7f90ac032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:04.447 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.445+0000 7f90bf7fe700 1 --2- 172.21.15.152:0/2229843420 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f90ac02ff60 0x7f90ac032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f90b00096a0 tx=0x7f90b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:04.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.449+0000 7f90bcff9700 1 -- 172.21.15.152:0/2229843420 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f90a80079c0 con 0x7f90c00f5410 2023-12-06T03:54:04.655 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.654+0000 7f90c6492700 1 -- 172.21.15.152:0/2229843420 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90c0045ad0 con 0x7f90c00f5410 2023-12-06T03:54:04.656 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.655+0000 7f90bcff9700 1 -- 172.21.15.152:0/2229843420 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f90a800f300 con 0x7f90c00f5410 2023-12-06T03:54:04.656 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:04.656 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:04.658 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.656+0000 7f90c6492700 1 -- 172.21.15.152:0/2229843420 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f90ac02ff60 msgr2=0x7f90ac032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:04.658 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.656+0000 7f90c6492700 1 --2- 172.21.15.152:0/2229843420 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f90ac02ff60 0x7f90ac032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f90b00096a0 tx=0x7f90b0006b40).stop 2023-12-06T03:54:04.658 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.656+0000 7f90c6492700 1 -- 172.21.15.152:0/2229843420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90c00f5410 msgr2=0x7f90c0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:04.658 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.656+0000 7f90c6492700 1 --2- 172.21.15.152:0/2229843420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90c00f5410 0x7f90c0109450 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f90a8014000 tx=0x7f90a800b040).stop 2023-12-06T03:54:04.658 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.656+0000 7f90c6492700 1 -- 172.21.15.152:0/2229843420 shutdown_connections 2023-12-06T03:54:04.658 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.656+0000 7f90c6492700 1 --2- 172.21.15.152:0/2229843420 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f90ac02ff60 0x7f90ac032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:04.658 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.656+0000 7f90c6492700 1 --2- 172.21.15.152:0/2229843420 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f90c00f5410 0x7f90c0109450 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:04.658 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.656+0000 7f90c6492700 1 -- 172.21.15.152:0/2229843420 >> 172.21.15.152:0/2229843420 conn(0x7f90c00f0b70 msgr2=0x7f90c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:04.658 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.656+0000 7f90c6492700 1 -- 172.21.15.152:0/2229843420 shutdown_connections 2023-12-06T03:54:04.658 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:04.656+0000 7f90c6492700 1 -- 172.21.15.152:0/2229843420 wait complete. 2023-12-06T03:54:04.659 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:06.029 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:06.030 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:06.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:04.390827+0000 mgr.smithi132.gokxja (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:06.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:04.655984+0000 mon.smithi132 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.152:0/2229843420' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:07.098 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.096+0000 7f3cf6cde700 1 -- 172.21.15.152:0/3112056223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf00f6510 msgr2=0x7f3cf00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:07.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.096+0000 7f3cf6cde700 1 --2- 172.21.15.152:0/3112056223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf00f6510 0x7f3cf00f6930 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f3ce4009fb0 tx=0x7f3ce4009b30).stop 2023-12-06T03:54:07.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.096+0000 7f3cf6cde700 1 -- 172.21.15.152:0/3112056223 shutdown_connections 2023-12-06T03:54:07.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.096+0000 7f3cf6cde700 1 --2- 172.21.15.152:0/3112056223 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf00f6510 0x7f3cf00f6930 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:07.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.096+0000 7f3cf6cde700 1 -- 172.21.15.152:0/3112056223 >> 172.21.15.152:0/3112056223 conn(0x7f3cf00f1c70 msgr2=0x7f3cf00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:07.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.096+0000 7f3cf6cde700 1 -- 172.21.15.152:0/3112056223 shutdown_connections 2023-12-06T03:54:07.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.096+0000 7f3cf6cde700 1 -- 172.21.15.152:0/3112056223 wait complete. 2023-12-06T03:54:07.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.096+0000 7f3cf6cde700 1 Processor -- start 2023-12-06T03:54:07.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.096+0000 7f3cf6cde700 1 -- start start 2023-12-06T03:54:07.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.097+0000 7f3cf6cde700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf00f6510 0x7f3cf006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:07.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.097+0000 7f3cf6cde700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf0069e20 con 0x7f3cf00f6510 2023-12-06T03:54:07.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.097+0000 7f3cf4a7a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf00f6510 0x7f3cf006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:07.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.097+0000 7f3cf4a7a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf00f6510 0x7f3cf006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:35746/0 (socket says 172.21.15.152:35746) 2023-12-06T03:54:07.100 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.097+0000 7f3cf4a7a700 1 -- 172.21.15.152:0/51820071 learned_addr learned my addr 172.21.15.152:0/51820071 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:07.101 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.097+0000 7f3cf4a7a700 1 -- 172.21.15.152:0/51820071 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce4005040 con 0x7f3cf00f6510 2023-12-06T03:54:07.101 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.097+0000 7f3cf4a7a700 1 --2- 172.21.15.152:0/51820071 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf00f6510 0x7f3cf006b9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f3ce4013510 tx=0x7f3ce400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:07.101 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.098+0000 7f3ced7fa700 1 -- 172.21.15.152:0/51820071 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce4009320 con 0x7f3cf00f6510 2023-12-06T03:54:07.101 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.098+0000 7f3cf6cde700 1 -- 172.21.15.152:0/51820071 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf0069fc0 con 0x7f3cf00f6510 2023-12-06T03:54:07.101 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.098+0000 7f3ced7fa700 1 -- 172.21.15.152:0/51820071 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ce4013f00 con 0x7f3cf00f6510 2023-12-06T03:54:07.101 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.098+0000 7f3ced7fa700 1 -- 172.21.15.152:0/51820071 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce4014d90 con 0x7f3cf00f6510 2023-12-06T03:54:07.101 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.098+0000 7f3cf6cde700 1 -- 172.21.15.152:0/51820071 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf006a450 con 0x7f3cf00f6510 2023-12-06T03:54:07.101 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.098+0000 7f3ced7fa700 1 -- 172.21.15.152:0/51820071 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ce4014420 con 0x7f3cf00f6510 2023-12-06T03:54:07.101 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.099+0000 7f3cf6cde700 1 -- 172.21.15.152:0/51820071 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf0059100 con 0x7f3cf00f6510 2023-12-06T03:54:07.101 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.099+0000 7f3ced7fa700 1 --2- 172.21.15.152:0/51820071 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce002ff60 0x7f3ce0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:07.101 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.099+0000 7f3ced7fa700 1 -- 172.21.15.152:0/51820071 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ce4041000 con 0x7f3cf00f6510 2023-12-06T03:54:07.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.099+0000 7f3ceffff700 1 --2- 172.21.15.152:0/51820071 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce002ff60 0x7f3ce0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:07.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.099+0000 7f3ceffff700 1 --2- 172.21.15.152:0/51820071 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce002ff60 0x7f3ce0032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f3cdc0096a0 tx=0x7f3cdc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:07.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.103+0000 7f3ced7fa700 1 -- 172.21.15.152:0/51820071 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ce40095f0 con 0x7f3cf00f6510 2023-12-06T03:54:07.310 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.308+0000 7f3cf6cde700 1 -- 172.21.15.152:0/51820071 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cf00fbf60 con 0x7f3cf00f6510 2023-12-06T03:54:07.310 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.309+0000 7f3ced7fa700 1 -- 172.21.15.152:0/51820071 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ce401db70 con 0x7f3cf00f6510 2023-12-06T03:54:07.311 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:07.311 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:07.312 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.311+0000 7f3cf6cde700 1 -- 172.21.15.152:0/51820071 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce002ff60 msgr2=0x7f3ce0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:07.312 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.311+0000 7f3cf6cde700 1 --2- 172.21.15.152:0/51820071 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce002ff60 0x7f3ce0032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f3cdc0096a0 tx=0x7f3cdc006b40).stop 2023-12-06T03:54:07.312 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.311+0000 7f3cf6cde700 1 -- 172.21.15.152:0/51820071 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf00f6510 msgr2=0x7f3cf006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:07.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.311+0000 7f3cf6cde700 1 --2- 172.21.15.152:0/51820071 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf00f6510 0x7f3cf006b9a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f3ce4013510 tx=0x7f3ce400b040).stop 2023-12-06T03:54:07.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.311+0000 7f3cf6cde700 1 -- 172.21.15.152:0/51820071 shutdown_connections 2023-12-06T03:54:07.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.311+0000 7f3cf6cde700 1 --2- 172.21.15.152:0/51820071 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce002ff60 0x7f3ce0032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:07.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.311+0000 7f3cf6cde700 1 --2- 172.21.15.152:0/51820071 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf00f6510 0x7f3cf006b9a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:07.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.311+0000 7f3cf6cde700 1 -- 172.21.15.152:0/51820071 >> 172.21.15.152:0/51820071 conn(0x7f3cf00f1c70 msgr2=0x7f3cf00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:07.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.311+0000 7f3cf6cde700 1 -- 172.21.15.152:0/51820071 shutdown_connections 2023-12-06T03:54:07.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:07.311+0000 7f3cf6cde700 1 -- 172.21.15.152:0/51820071 wait complete. 2023-12-06T03:54:07.313 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:08.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:07 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:06.391226+0000 mgr.smithi132.gokxja (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:08.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:07 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:07.310257+0000 mon.smithi132 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.152:0/51820071' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:08.686 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:08.686 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:09.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.922+0000 7f047c51c700 1 -- 172.21.15.152:0/2108069487 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04740f5410 msgr2=0x7f04740f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:09.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.922+0000 7f047c51c700 1 --2- 172.21.15.152:0/2108069487 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04740f5410 0x7f04740f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f0464009fb0 tx=0x7f0464009b30).stop 2023-12-06T03:54:09.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.922+0000 7f047c51c700 1 -- 172.21.15.152:0/2108069487 shutdown_connections 2023-12-06T03:54:09.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.922+0000 7f047c51c700 1 --2- 172.21.15.152:0/2108069487 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04740f5410 0x7f04740f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:09.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.922+0000 7f047c51c700 1 -- 172.21.15.152:0/2108069487 >> 172.21.15.152:0/2108069487 conn(0x7f04740f0b70 msgr2=0x7f04740f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:09.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.922+0000 7f047c51c700 1 -- 172.21.15.152:0/2108069487 shutdown_connections 2023-12-06T03:54:09.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.922+0000 7f047c51c700 1 -- 172.21.15.152:0/2108069487 wait complete. 2023-12-06T03:54:09.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.923+0000 7f047c51c700 1 Processor -- start 2023-12-06T03:54:09.926 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.923+0000 7f047c51c700 1 -- start start 2023-12-06T03:54:09.926 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.923+0000 7f047c51c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04740f5410 0x7f0474105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:09.926 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.923+0000 7f047c51c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0474105b20 con 0x7f04740f5410 2023-12-06T03:54:09.926 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.923+0000 7f047a2b8700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04740f5410 0x7f0474105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:09.926 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.923+0000 7f047a2b8700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04740f5410 0x7f0474105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:33986/0 (socket says 172.21.15.152:33986) 2023-12-06T03:54:09.926 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.923+0000 7f047a2b8700 1 -- 172.21.15.152:0/908262686 learned_addr learned my addr 172.21.15.152:0/908262686 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:09.927 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.924+0000 7f047a2b8700 1 -- 172.21.15.152:0/908262686 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0464005040 con 0x7f04740f5410 2023-12-06T03:54:09.927 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.924+0000 7f047a2b8700 1 --2- 172.21.15.152:0/908262686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04740f5410 0x7f0474105630 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f0464013510 tx=0x7f046400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:09.927 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.924+0000 7f046affd700 1 -- 172.21.15.152:0/908262686 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0464009320 con 0x7f04740f5410 2023-12-06T03:54:09.927 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.924+0000 7f046affd700 1 -- 172.21.15.152:0/908262686 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0464013f00 con 0x7f04740f5410 2023-12-06T03:54:09.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.924+0000 7f047c51c700 1 -- 172.21.15.152:0/908262686 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0474105d20 con 0x7f04740f5410 2023-12-06T03:54:09.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.924+0000 7f046affd700 1 -- 172.21.15.152:0/908262686 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0464014d90 con 0x7f04740f5410 2023-12-06T03:54:09.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.924+0000 7f047c51c700 1 -- 172.21.15.152:0/908262686 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f04741061b0 con 0x7f04740f5410 2023-12-06T03:54:09.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.925+0000 7f046affd700 1 -- 172.21.15.152:0/908262686 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04640095f0 con 0x7f04740f5410 2023-12-06T03:54:09.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.925+0000 7f047c51c700 1 -- 172.21.15.152:0/908262686 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0474045ad0 con 0x7f04740f5410 2023-12-06T03:54:09.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.925+0000 7f046affd700 1 --2- 172.21.15.152:0/908262686 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f046002ff60 0x7f0460032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:09.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.925+0000 7f046affd700 1 -- 172.21.15.152:0/908262686 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0464041080 con 0x7f04740f5410 2023-12-06T03:54:09.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.925+0000 7f0479ab7700 1 --2- 172.21.15.152:0/908262686 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f046002ff60 0x7f0460032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:09.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.926+0000 7f0479ab7700 1 --2- 172.21.15.152:0/908262686 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f046002ff60 0x7f0460032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f04700096a0 tx=0x7f0470006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:09.931 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:09.929+0000 7f046affd700 1 -- 172.21.15.152:0/908262686 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f046401d7f0 con 0x7f04740f5410 2023-12-06T03:54:10.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:09 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:08.391558+0000 mgr.smithi132.gokxja (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:10.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:10.132+0000 7f047c51c700 1 -- 172.21.15.152:0/908262686 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04740fae60 con 0x7f04740f5410 2023-12-06T03:54:10.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:10.133+0000 7f046affd700 1 -- 172.21.15.152:0/908262686 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0464010230 con 0x7f04740f5410 2023-12-06T03:54:10.134 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:10.134 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:10.134+0000 7f047c51c700 1 -- 172.21.15.152:0/908262686 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f046002ff60 msgr2=0x7f0460032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:10.134+0000 7f047c51c700 1 --2- 172.21.15.152:0/908262686 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f046002ff60 0x7f0460032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f04700096a0 tx=0x7f0470006b40).stop 2023-12-06T03:54:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:10.134+0000 7f047c51c700 1 -- 172.21.15.152:0/908262686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04740f5410 msgr2=0x7f0474105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:10.134+0000 7f047c51c700 1 --2- 172.21.15.152:0/908262686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04740f5410 0x7f0474105630 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f0464013510 tx=0x7f046400b040).stop 2023-12-06T03:54:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:10.134+0000 7f047c51c700 1 -- 172.21.15.152:0/908262686 shutdown_connections 2023-12-06T03:54:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:10.134+0000 7f047c51c700 1 --2- 172.21.15.152:0/908262686 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f046002ff60 0x7f0460032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:10.134+0000 7f047c51c700 1 --2- 172.21.15.152:0/908262686 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f04740f5410 0x7f0474105630 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:10.134+0000 7f047c51c700 1 -- 172.21.15.152:0/908262686 >> 172.21.15.152:0/908262686 conn(0x7f04740f0b70 msgr2=0x7f04740f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:10.134+0000 7f047c51c700 1 -- 172.21.15.152:0/908262686 shutdown_connections 2023-12-06T03:54:10.137 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:10.134+0000 7f047c51c700 1 -- 172.21.15.152:0/908262686 wait complete. 2023-12-06T03:54:10.137 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:11.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:10 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:10.134033+0000 mon.smithi132 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.152:0/908262686' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:11.518 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:11.522 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:12.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:11 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:10.391946+0000 mgr.smithi132.gokxja (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:12.639 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.637+0000 7f72d18c3700 1 -- 172.21.15.152:0/2834267363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72cc060280 msgr2=0x7f72cc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:12.639 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.637+0000 7f72d18c3700 1 --2- 172.21.15.152:0/2834267363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72cc060280 0x7f72cc0f90f0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f72bc009fb0 tx=0x7f72bc009b30).stop 2023-12-06T03:54:12.639 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.637+0000 7f72d18c3700 1 -- 172.21.15.152:0/2834267363 shutdown_connections 2023-12-06T03:54:12.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.637+0000 7f72d18c3700 1 --2- 172.21.15.152:0/2834267363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72cc060280 0x7f72cc0f90f0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:12.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.637+0000 7f72d18c3700 1 -- 172.21.15.152:0/2834267363 >> 172.21.15.152:0/2834267363 conn(0x7f72cc0f0b70 msgr2=0x7f72cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:12.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.637+0000 7f72d18c3700 1 -- 172.21.15.152:0/2834267363 shutdown_connections 2023-12-06T03:54:12.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.637+0000 7f72d18c3700 1 -- 172.21.15.152:0/2834267363 wait complete. 2023-12-06T03:54:12.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.638+0000 7f72d18c3700 1 Processor -- start 2023-12-06T03:54:12.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.638+0000 7f72d18c3700 1 -- start start 2023-12-06T03:54:12.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.638+0000 7f72d18c3700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72cc060280 0x7f72cc1066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:12.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.638+0000 7f72d18c3700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72cc106bb0 con 0x7f72cc060280 2023-12-06T03:54:12.642 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.638+0000 7f72caffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72cc060280 0x7f72cc1066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:12.642 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.638+0000 7f72caffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72cc060280 0x7f72cc1066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:34014/0 (socket says 172.21.15.152:34014) 2023-12-06T03:54:12.642 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.638+0000 7f72caffd700 1 -- 172.21.15.152:0/3639606089 learned_addr learned my addr 172.21.15.152:0/3639606089 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:12.642 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.638+0000 7f72caffd700 1 -- 172.21.15.152:0/3639606089 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72bc005040 con 0x7f72cc060280 2023-12-06T03:54:12.642 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.639+0000 7f72caffd700 1 --2- 172.21.15.152:0/3639606089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72cc060280 0x7f72cc1066c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f72bc013510 tx=0x7f72bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:12.642 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.639+0000 7f72b37fe700 1 -- 172.21.15.152:0/3639606089 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72bc009320 con 0x7f72cc060280 2023-12-06T03:54:12.642 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.639+0000 7f72d18c3700 1 -- 172.21.15.152:0/3639606089 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72cc106db0 con 0x7f72cc060280 2023-12-06T03:54:12.642 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.639+0000 7f72b37fe700 1 -- 172.21.15.152:0/3639606089 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72bc013f00 con 0x7f72cc060280 2023-12-06T03:54:12.642 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.639+0000 7f72b37fe700 1 -- 172.21.15.152:0/3639606089 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72bc014dc0 con 0x7f72cc060280 2023-12-06T03:54:12.642 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.639+0000 7f72d18c3700 1 -- 172.21.15.152:0/3639606089 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72cc107240 con 0x7f72cc060280 2023-12-06T03:54:12.643 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.639+0000 7f72b37fe700 1 -- 172.21.15.152:0/3639606089 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72bc01d3d0 con 0x7f72cc060280 2023-12-06T03:54:12.643 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.640+0000 7f72b37fe700 1 --2- 172.21.15.152:0/3639606089 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f72b402ff60 0x7f72b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:12.643 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.640+0000 7f72b37fe700 1 -- 172.21.15.152:0/3639606089 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72bc0410d0 con 0x7f72cc060280 2023-12-06T03:54:12.643 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.640+0000 7f72d18c3700 1 -- 172.21.15.152:0/3639606089 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72cc045ad0 con 0x7f72cc060280 2023-12-06T03:54:12.643 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.640+0000 7f72ca7fc700 1 --2- 172.21.15.152:0/3639606089 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f72b402ff60 0x7f72b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:12.643 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.640+0000 7f72ca7fc700 1 --2- 172.21.15.152:0/3639606089 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f72b402ff60 0x7f72b4032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f72c00096a0 tx=0x7f72c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:12.646 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.644+0000 7f72b37fe700 1 -- 172.21.15.152:0/3639606089 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72bc026490 con 0x7f72cc060280 2023-12-06T03:54:12.854 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.853+0000 7f72d18c3700 1 -- 172.21.15.152:0/3639606089 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72cc107f80 con 0x7f72cc060280 2023-12-06T03:54:12.855 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.853+0000 7f72b37fe700 1 -- 172.21.15.152:0/3639606089 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f72bc0126f0 con 0x7f72cc060280 2023-12-06T03:54:12.855 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:12.855 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:12.857 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.855+0000 7f72d18c3700 1 -- 172.21.15.152:0/3639606089 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f72b402ff60 msgr2=0x7f72b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:12.857 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.855+0000 7f72d18c3700 1 --2- 172.21.15.152:0/3639606089 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f72b402ff60 0x7f72b4032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f72c00096a0 tx=0x7f72c0006b40).stop 2023-12-06T03:54:12.857 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.855+0000 7f72d18c3700 1 -- 172.21.15.152:0/3639606089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72cc060280 msgr2=0x7f72cc1066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:12.857 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.855+0000 7f72d18c3700 1 --2- 172.21.15.152:0/3639606089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72cc060280 0x7f72cc1066c0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f72bc013510 tx=0x7f72bc00b040).stop 2023-12-06T03:54:12.857 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.855+0000 7f72d18c3700 1 -- 172.21.15.152:0/3639606089 shutdown_connections 2023-12-06T03:54:12.857 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.855+0000 7f72d18c3700 1 --2- 172.21.15.152:0/3639606089 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f72b402ff60 0x7f72b4032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:12.857 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.855+0000 7f72d18c3700 1 --2- 172.21.15.152:0/3639606089 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f72cc060280 0x7f72cc1066c0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:12.857 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.855+0000 7f72d18c3700 1 -- 172.21.15.152:0/3639606089 >> 172.21.15.152:0/3639606089 conn(0x7f72cc0f0b70 msgr2=0x7f72cc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:12.858 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.855+0000 7f72d18c3700 1 -- 172.21.15.152:0/3639606089 shutdown_connections 2023-12-06T03:54:12.858 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:12.855+0000 7f72d18c3700 1 -- 172.21.15.152:0/3639606089 wait complete. 2023-12-06T03:54:12.858 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:14.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:13 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:12.392228+0000 mgr.smithi132.gokxja (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:14.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:13 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:12.854869+0000 mon.smithi132 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.152:0/3639606089' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:14.235 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:14.235 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:15.346 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.344+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3968541022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cd80f5410 msgr2=0x7f8cd80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:15.346 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.344+0000 7f8cdd73e700 1 --2- 172.21.15.152:0/3968541022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cd80f5410 0x7f8cd80f5830 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f8cc8009fb0 tx=0x7f8cc8009b30).stop 2023-12-06T03:54:15.346 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.344+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3968541022 shutdown_connections 2023-12-06T03:54:15.346 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.344+0000 7f8cdd73e700 1 --2- 172.21.15.152:0/3968541022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cd80f5410 0x7f8cd80f5830 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:15.346 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.344+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3968541022 >> 172.21.15.152:0/3968541022 conn(0x7f8cd80f0b70 msgr2=0x7f8cd80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:15.347 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.344+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3968541022 shutdown_connections 2023-12-06T03:54:15.347 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.344+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3968541022 wait complete. 2023-12-06T03:54:15.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.345+0000 7f8cdd73e700 1 Processor -- start 2023-12-06T03:54:15.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.345+0000 7f8cdd73e700 1 -- start start 2023-12-06T03:54:15.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.345+0000 7f8cdd73e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cd80f5410 0x7f8cd81055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:15.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.345+0000 7f8cdd73e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd8105ac0 con 0x7f8cd80f5410 2023-12-06T03:54:15.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.346+0000 7f8cd6ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cd80f5410 0x7f8cd81055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:15.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.346+0000 7f8cd6ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cd80f5410 0x7f8cd81055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:34032/0 (socket says 172.21.15.152:34032) 2023-12-06T03:54:15.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.346+0000 7f8cd6ffd700 1 -- 172.21.15.152:0/3328804015 learned_addr learned my addr 172.21.15.152:0/3328804015 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:15.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.346+0000 7f8cd6ffd700 1 -- 172.21.15.152:0/3328804015 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc8005040 con 0x7f8cd80f5410 2023-12-06T03:54:15.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.346+0000 7f8cd6ffd700 1 --2- 172.21.15.152:0/3328804015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cd80f5410 0x7f8cd81055d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f8cc8013510 tx=0x7f8cc800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:15.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.346+0000 7f8cbf7fe700 1 -- 172.21.15.152:0/3328804015 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc8009320 con 0x7f8cd80f5410 2023-12-06T03:54:15.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.346+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3328804015 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd8105cc0 con 0x7f8cd80f5410 2023-12-06T03:54:15.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.346+0000 7f8cbf7fe700 1 -- 172.21.15.152:0/3328804015 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cc8013f00 con 0x7f8cd80f5410 2023-12-06T03:54:15.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.346+0000 7f8cbf7fe700 1 -- 172.21.15.152:0/3328804015 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cc8014d90 con 0x7f8cd80f5410 2023-12-06T03:54:15.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.346+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3328804015 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd8106150 con 0x7f8cd80f5410 2023-12-06T03:54:15.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.347+0000 7f8cbf7fe700 1 -- 172.21.15.152:0/3328804015 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8cc8014420 con 0x7f8cd80f5410 2023-12-06T03:54:15.350 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.347+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3328804015 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd8045ad0 con 0x7f8cd80f5410 2023-12-06T03:54:15.350 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.347+0000 7f8cbf7fe700 1 --2- 172.21.15.152:0/3328804015 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8cc002ff60 0x7f8cc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:15.350 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.347+0000 7f8cbf7fe700 1 -- 172.21.15.152:0/3328804015 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8cc8040fb0 con 0x7f8cd80f5410 2023-12-06T03:54:15.350 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.348+0000 7f8cd67fc700 1 --2- 172.21.15.152:0/3328804015 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8cc002ff60 0x7f8cc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:15.350 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.348+0000 7f8cd67fc700 1 --2- 172.21.15.152:0/3328804015 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8cc002ff60 0x7f8cc0032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f8ccc0096a0 tx=0x7f8ccc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:15.353 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.352+0000 7f8cbf7fe700 1 -- 172.21.15.152:0/3328804015 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cc80095f0 con 0x7f8cd80f5410 2023-12-06T03:54:15.559 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.557+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3328804015 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8cd8107000 con 0x7f8cd80f5410 2023-12-06T03:54:15.559 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.558+0000 7f8cbf7fe700 1 -- 172.21.15.152:0/3328804015 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8cc801db70 con 0x7f8cd80f5410 2023-12-06T03:54:15.560 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:15.560 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:15.561 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.559+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3328804015 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8cc002ff60 msgr2=0x7f8cc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:15.561 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.559+0000 7f8cdd73e700 1 --2- 172.21.15.152:0/3328804015 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8cc002ff60 0x7f8cc0032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f8ccc0096a0 tx=0x7f8ccc006b40).stop 2023-12-06T03:54:15.561 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.559+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3328804015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cd80f5410 msgr2=0x7f8cd81055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:15.561 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.559+0000 7f8cdd73e700 1 --2- 172.21.15.152:0/3328804015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cd80f5410 0x7f8cd81055d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f8cc8013510 tx=0x7f8cc800b040).stop 2023-12-06T03:54:15.561 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.559+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3328804015 shutdown_connections 2023-12-06T03:54:15.561 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.559+0000 7f8cdd73e700 1 --2- 172.21.15.152:0/3328804015 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8cc002ff60 0x7f8cc0032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:15.562 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.559+0000 7f8cdd73e700 1 --2- 172.21.15.152:0/3328804015 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8cd80f5410 0x7f8cd81055d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:15.562 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.559+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3328804015 >> 172.21.15.152:0/3328804015 conn(0x7f8cd80f0b70 msgr2=0x7f8cd80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:15.562 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.559+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3328804015 shutdown_connections 2023-12-06T03:54:15.562 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:15.560+0000 7f8cdd73e700 1 -- 172.21.15.152:0/3328804015 wait complete. 2023-12-06T03:54:15.562 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:16.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:15 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:14.392568+0000 mgr.smithi132.gokxja (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:16.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:15 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:15.559181+0000 mon.smithi132 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.152:0/3328804015' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:16.935 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:16.935 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:18.075 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.073+0000 7fd6022f8700 1 -- 172.21.15.152:0/4166532413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5fc060200 msgr2=0x7fd5fc0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:18.075 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.073+0000 7fd6022f8700 1 --2- 172.21.15.152:0/4166532413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5fc060200 0x7fd5fc0fb570 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fd5ec009fb0 tx=0x7fd5ec009b30).stop 2023-12-06T03:54:18.075 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.073+0000 7fd6022f8700 1 -- 172.21.15.152:0/4166532413 shutdown_connections 2023-12-06T03:54:18.075 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.073+0000 7fd6022f8700 1 --2- 172.21.15.152:0/4166532413 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5fc060200 0x7fd5fc0fb570 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:18.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.073+0000 7fd6022f8700 1 -- 172.21.15.152:0/4166532413 >> 172.21.15.152:0/4166532413 conn(0x7fd5fc0f0b70 msgr2=0x7fd5fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:18.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.073+0000 7fd6022f8700 1 -- 172.21.15.152:0/4166532413 shutdown_connections 2023-12-06T03:54:18.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.073+0000 7fd6022f8700 1 -- 172.21.15.152:0/4166532413 wait complete. 2023-12-06T03:54:18.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.073+0000 7fd6022f8700 1 Processor -- start 2023-12-06T03:54:18.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.073+0000 7fd6022f8700 1 -- start start 2023-12-06T03:54:18.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.074+0000 7fd6022f8700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5fc060200 0x7fd5fc1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:18.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.074+0000 7fd6022f8700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5fc108dc0 con 0x7fd5fc060200 2023-12-06T03:54:18.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.074+0000 7fd5fbfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5fc060200 0x7fd5fc1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:18.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.074+0000 7fd5fbfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5fc060200 0x7fd5fc1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:34046/0 (socket says 172.21.15.152:34046) 2023-12-06T03:54:18.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.074+0000 7fd5fbfff700 1 -- 172.21.15.152:0/105775826 learned_addr learned my addr 172.21.15.152:0/105775826 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:18.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.074+0000 7fd5fbfff700 1 -- 172.21.15.152:0/105775826 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5ec005040 con 0x7fd5fc060200 2023-12-06T03:54:18.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.074+0000 7fd5fbfff700 1 --2- 172.21.15.152:0/105775826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5fc060200 0x7fd5fc1088d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fd5ec013510 tx=0x7fd5ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:18.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.074+0000 7fd5f8ff9700 1 -- 172.21.15.152:0/105775826 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5ec009320 con 0x7fd5fc060200 2023-12-06T03:54:18.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.074+0000 7fd5f8ff9700 1 -- 172.21.15.152:0/105775826 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd5ec013f00 con 0x7fd5fc060200 2023-12-06T03:54:18.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.074+0000 7fd6022f8700 1 -- 172.21.15.152:0/105775826 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5fc108fc0 con 0x7fd5fc060200 2023-12-06T03:54:18.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.074+0000 7fd5f8ff9700 1 -- 172.21.15.152:0/105775826 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5ec014dc0 con 0x7fd5fc060200 2023-12-06T03:54:18.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.074+0000 7fd6022f8700 1 -- 172.21.15.152:0/105775826 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5fc109450 con 0x7fd5fc060200 2023-12-06T03:54:18.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.075+0000 7fd5f8ff9700 1 -- 172.21.15.152:0/105775826 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd5ec0095f0 con 0x7fd5fc060200 2023-12-06T03:54:18.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.075+0000 7fd6022f8700 1 -- 172.21.15.152:0/105775826 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5fc045ad0 con 0x7fd5fc060200 2023-12-06T03:54:18.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.075+0000 7fd5f8ff9700 1 --2- 172.21.15.152:0/105775826 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd5e802ff60 0x7fd5e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:18.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.075+0000 7fd5f8ff9700 1 -- 172.21.15.152:0/105775826 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd5ec0410c0 con 0x7fd5fc060200 2023-12-06T03:54:18.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.076+0000 7fd5fb7fe700 1 --2- 172.21.15.152:0/105775826 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd5e802ff60 0x7fd5e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:18.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.076+0000 7fd5fb7fe700 1 --2- 172.21.15.152:0/105775826 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd5e802ff60 0x7fd5e8032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fd5e40096a0 tx=0x7fd5e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:18.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.080+0000 7fd5f8ff9700 1 -- 172.21.15.152:0/105775826 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd5ec014420 con 0x7fd5fc060200 2023-12-06T03:54:18.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:17 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:16.392833+0000 mgr.smithi132.gokxja (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:18.289 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.287+0000 7fd6022f8700 1 -- 172.21.15.152:0/105775826 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5fc10a300 con 0x7fd5fc060200 2023-12-06T03:54:18.290 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.288+0000 7fd5f8ff9700 1 -- 172.21.15.152:0/105775826 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd5ec010230 con 0x7fd5fc060200 2023-12-06T03:54:18.290 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:18.290 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:18.291 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.290+0000 7fd6022f8700 1 -- 172.21.15.152:0/105775826 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd5e802ff60 msgr2=0x7fd5e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:18.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.290+0000 7fd6022f8700 1 --2- 172.21.15.152:0/105775826 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd5e802ff60 0x7fd5e8032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fd5e40096a0 tx=0x7fd5e4006b40).stop 2023-12-06T03:54:18.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.290+0000 7fd6022f8700 1 -- 172.21.15.152:0/105775826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5fc060200 msgr2=0x7fd5fc1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:18.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.290+0000 7fd6022f8700 1 --2- 172.21.15.152:0/105775826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5fc060200 0x7fd5fc1088d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fd5ec013510 tx=0x7fd5ec00b040).stop 2023-12-06T03:54:18.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.290+0000 7fd6022f8700 1 -- 172.21.15.152:0/105775826 shutdown_connections 2023-12-06T03:54:18.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.290+0000 7fd6022f8700 1 --2- 172.21.15.152:0/105775826 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd5e802ff60 0x7fd5e8032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:18.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.290+0000 7fd6022f8700 1 --2- 172.21.15.152:0/105775826 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd5fc060200 0x7fd5fc1088d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:18.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.290+0000 7fd6022f8700 1 -- 172.21.15.152:0/105775826 >> 172.21.15.152:0/105775826 conn(0x7fd5fc0f0b70 msgr2=0x7fd5fc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:18.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.290+0000 7fd6022f8700 1 -- 172.21.15.152:0/105775826 shutdown_connections 2023-12-06T03:54:18.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:18.290+0000 7fd6022f8700 1 -- 172.21.15.152:0/105775826 wait complete. 2023-12-06T03:54:18.292 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:19.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:18 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:18.289453+0000 mon.smithi132 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.152:0/105775826' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:19.663 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:19.664 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:20.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:19 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:18.393202+0000 mgr.smithi132.gokxja (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:20.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.869+0000 7f0fbc883700 1 -- 172.21.15.152:0/3681854728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fb80f5330 msgr2=0x7f0fb80f5750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:20.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.869+0000 7f0fbc883700 1 --2- 172.21.15.152:0/3681854728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fb80f5330 0x7f0fb80f5750 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f0fac009fb0 tx=0x7f0fac009b30).stop 2023-12-06T03:54:20.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.869+0000 7f0fbc883700 1 -- 172.21.15.152:0/3681854728 shutdown_connections 2023-12-06T03:54:20.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.869+0000 7f0fbc883700 1 --2- 172.21.15.152:0/3681854728 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fb80f5330 0x7f0fb80f5750 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:20.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.869+0000 7f0fbc883700 1 -- 172.21.15.152:0/3681854728 >> 172.21.15.152:0/3681854728 conn(0x7f0fb80f0a90 msgr2=0x7f0fb80f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:20.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.869+0000 7f0fbc883700 1 -- 172.21.15.152:0/3681854728 shutdown_connections 2023-12-06T03:54:20.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.869+0000 7f0fbc883700 1 -- 172.21.15.152:0/3681854728 wait complete. 2023-12-06T03:54:20.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.870+0000 7f0fbc883700 1 Processor -- start 2023-12-06T03:54:20.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.870+0000 7f0fbc883700 1 -- start start 2023-12-06T03:54:20.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.870+0000 7f0fbc883700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fb80f5330 0x7f0fb81096d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:20.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.870+0000 7f0fbc883700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0fb8060240 con 0x7f0fb80f5330 2023-12-06T03:54:20.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.870+0000 7f0fb659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fb80f5330 0x7f0fb81096d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:20.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.870+0000 7f0fb659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fb80f5330 0x7f0fb81096d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:39156/0 (socket says 172.21.15.152:39156) 2023-12-06T03:54:20.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.870+0000 7f0fb659c700 1 -- 172.21.15.152:0/2106474975 learned_addr learned my addr 172.21.15.152:0/2106474975 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:20.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.871+0000 7f0fb659c700 1 -- 172.21.15.152:0/2106474975 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0fac005040 con 0x7f0fb80f5330 2023-12-06T03:54:20.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.871+0000 7f0fb659c700 1 --2- 172.21.15.152:0/2106474975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fb80f5330 0x7f0fb81096d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f0fac013510 tx=0x7f0fac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:20.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.871+0000 7f0fa6ffd700 1 -- 172.21.15.152:0/2106474975 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fac009320 con 0x7f0fb80f5330 2023-12-06T03:54:20.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.871+0000 7f0fbc883700 1 -- 172.21.15.152:0/2106474975 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0fb8060440 con 0x7f0fb80f5330 2023-12-06T03:54:20.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.871+0000 7f0fa6ffd700 1 -- 172.21.15.152:0/2106474975 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0fac013f00 con 0x7f0fb80f5330 2023-12-06T03:54:20.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.871+0000 7f0fa6ffd700 1 -- 172.21.15.152:0/2106474975 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0fac014d90 con 0x7f0fb80f5330 2023-12-06T03:54:20.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.871+0000 7f0fbc883700 1 -- 172.21.15.152:0/2106474975 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0fb8109d80 con 0x7f0fb80f5330 2023-12-06T03:54:20.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.872+0000 7f0fa6ffd700 1 -- 172.21.15.152:0/2106474975 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0fac014420 con 0x7f0fb80f5330 2023-12-06T03:54:20.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.872+0000 7f0fbc883700 1 -- 172.21.15.152:0/2106474975 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0fb8045ad0 con 0x7f0fb80f5330 2023-12-06T03:54:20.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.872+0000 7f0fa6ffd700 1 --2- 172.21.15.152:0/2106474975 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f0fa002ff60 0x7f0fa0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:20.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.872+0000 7f0fa6ffd700 1 -- 172.21.15.152:0/2106474975 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0fac041070 con 0x7f0fb80f5330 2023-12-06T03:54:20.876 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.872+0000 7f0fb5d9b700 1 --2- 172.21.15.152:0/2106474975 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f0fa002ff60 0x7f0fa0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:20.876 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.873+0000 7f0fb5d9b700 1 --2- 172.21.15.152:0/2106474975 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f0fa002ff60 0x7f0fa0032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f0fa80096a0 tx=0x7f0fa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:20.878 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:20.877+0000 7f0fa6ffd700 1 -- 172.21.15.152:0/2106474975 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0fac0095f0 con 0x7f0fb80f5330 2023-12-06T03:54:21.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:21.081+0000 7f0fbc883700 1 -- 172.21.15.152:0/2106474975 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0fb805ea50 con 0x7f0fb80f5330 2023-12-06T03:54:21.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:21.082+0000 7f0fa6ffd700 1 -- 172.21.15.152:0/2106474975 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0fac01db70 con 0x7f0fb80f5330 2023-12-06T03:54:21.084 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:21.084 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:21.085 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:21.084+0000 7f0fbc883700 1 -- 172.21.15.152:0/2106474975 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f0fa002ff60 msgr2=0x7f0fa0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:21.085 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:21.084+0000 7f0fbc883700 1 --2- 172.21.15.152:0/2106474975 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f0fa002ff60 0x7f0fa0032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f0fa80096a0 tx=0x7f0fa8006b40).stop 2023-12-06T03:54:21.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:21.084+0000 7f0fbc883700 1 -- 172.21.15.152:0/2106474975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fb80f5330 msgr2=0x7f0fb81096d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:21.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:21.084+0000 7f0fbc883700 1 --2- 172.21.15.152:0/2106474975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fb80f5330 0x7f0fb81096d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f0fac013510 tx=0x7f0fac00b040).stop 2023-12-06T03:54:21.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:21.084+0000 7f0fbc883700 1 -- 172.21.15.152:0/2106474975 shutdown_connections 2023-12-06T03:54:21.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:21.084+0000 7f0fbc883700 1 --2- 172.21.15.152:0/2106474975 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f0fa002ff60 0x7f0fa0032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:21.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:21.084+0000 7f0fbc883700 1 --2- 172.21.15.152:0/2106474975 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f0fb80f5330 0x7f0fb81096d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:21.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:21.084+0000 7f0fbc883700 1 -- 172.21.15.152:0/2106474975 >> 172.21.15.152:0/2106474975 conn(0x7f0fb80f0a90 msgr2=0x7f0fb80f1530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:21.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:21.084+0000 7f0fbc883700 1 -- 172.21.15.152:0/2106474975 shutdown_connections 2023-12-06T03:54:21.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:21.084+0000 7f0fbc883700 1 -- 172.21.15.152:0/2106474975 wait complete. 2023-12-06T03:54:21.086 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:22.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:20.393480+0000 mgr.smithi132.gokxja (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:22.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:21.083294+0000 mon.smithi132 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.152:0/2106474975' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:22.469 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:22.470 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:23.539 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.536+0000 7f2db484d700 1 -- 172.21.15.152:0/4294237525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2db00f6510 msgr2=0x7f2db00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:23.539 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.536+0000 7f2db484d700 1 --2- 172.21.15.152:0/4294237525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2db00f6510 0x7f2db00f6930 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f2da4009fb0 tx=0x7f2da4009b30).stop 2023-12-06T03:54:23.539 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.536+0000 7f2db484d700 1 -- 172.21.15.152:0/4294237525 shutdown_connections 2023-12-06T03:54:23.539 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.536+0000 7f2db484d700 1 --2- 172.21.15.152:0/4294237525 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2db00f6510 0x7f2db00f6930 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:23.539 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.536+0000 7f2db484d700 1 -- 172.21.15.152:0/4294237525 >> 172.21.15.152:0/4294237525 conn(0x7f2db00f1c70 msgr2=0x7f2db00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:23.540 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.536+0000 7f2db484d700 1 -- 172.21.15.152:0/4294237525 shutdown_connections 2023-12-06T03:54:23.540 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.536+0000 7f2db484d700 1 -- 172.21.15.152:0/4294237525 wait complete. 2023-12-06T03:54:23.540 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.536+0000 7f2db484d700 1 Processor -- start 2023-12-06T03:54:23.540 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.536+0000 7f2db484d700 1 -- start start 2023-12-06T03:54:23.540 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.537+0000 7f2db484d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2db00f6510 0x7f2db0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:23.540 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.537+0000 7f2db484d700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2db0106c20 con 0x7f2db00f6510 2023-12-06T03:54:23.540 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.537+0000 7f2dae59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2db00f6510 0x7f2db0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:23.540 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.537+0000 7f2dae59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2db00f6510 0x7f2db0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:39176/0 (socket says 172.21.15.152:39176) 2023-12-06T03:54:23.540 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.537+0000 7f2dae59c700 1 -- 172.21.15.152:0/991375438 learned_addr learned my addr 172.21.15.152:0/991375438 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:23.540 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.537+0000 7f2dae59c700 1 -- 172.21.15.152:0/991375438 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2da4005040 con 0x7f2db00f6510 2023-12-06T03:54:23.540 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.537+0000 7f2dae59c700 1 --2- 172.21.15.152:0/991375438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2db00f6510 0x7f2db0106730 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f2da4013510 tx=0x7f2da400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:23.541 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.537+0000 7f2d9effd700 1 -- 172.21.15.152:0/991375438 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2da4009320 con 0x7f2db00f6510 2023-12-06T03:54:23.541 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.537+0000 7f2db484d700 1 -- 172.21.15.152:0/991375438 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2db0106e20 con 0x7f2db00f6510 2023-12-06T03:54:23.541 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.537+0000 7f2d9effd700 1 -- 172.21.15.152:0/991375438 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2da4013f00 con 0x7f2db00f6510 2023-12-06T03:54:23.541 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.537+0000 7f2d9effd700 1 -- 172.21.15.152:0/991375438 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2da4014d90 con 0x7f2db00f6510 2023-12-06T03:54:23.541 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.537+0000 7f2db484d700 1 -- 172.21.15.152:0/991375438 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2db01072b0 con 0x7f2db00f6510 2023-12-06T03:54:23.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.538+0000 7f2d9effd700 1 -- 172.21.15.152:0/991375438 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2da40095f0 con 0x7f2db00f6510 2023-12-06T03:54:23.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.538+0000 7f2db484d700 1 -- 172.21.15.152:0/991375438 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2db0045ad0 con 0x7f2db00f6510 2023-12-06T03:54:23.542 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.538+0000 7f2d9effd700 1 --2- 172.21.15.152:0/991375438 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2d9802ff60 0x7f2d98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:23.543 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.538+0000 7f2d9effd700 1 -- 172.21.15.152:0/991375438 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2da40410c0 con 0x7f2db00f6510 2023-12-06T03:54:23.543 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.539+0000 7f2dadd9b700 1 --2- 172.21.15.152:0/991375438 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2d9802ff60 0x7f2d98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:23.543 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.539+0000 7f2dadd9b700 1 --2- 172.21.15.152:0/991375438 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2d9802ff60 0x7f2d98032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f2da00096a0 tx=0x7f2da0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:23.544 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.543+0000 7f2d9effd700 1 -- 172.21.15.152:0/991375438 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2da4014420 con 0x7f2db00f6510 2023-12-06T03:54:23.754 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.753+0000 7f2db484d700 1 -- 172.21.15.152:0/991375438 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2db00fbf60 con 0x7f2db00f6510 2023-12-06T03:54:23.755 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.753+0000 7f2d9effd700 1 -- 172.21.15.152:0/991375438 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2da4010230 con 0x7f2db00f6510 2023-12-06T03:54:23.755 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:23.755 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:23.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.755+0000 7f2db484d700 1 -- 172.21.15.152:0/991375438 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2d9802ff60 msgr2=0x7f2d98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:23.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.755+0000 7f2db484d700 1 --2- 172.21.15.152:0/991375438 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2d9802ff60 0x7f2d98032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f2da00096a0 tx=0x7f2da0006b40).stop 2023-12-06T03:54:23.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.755+0000 7f2db484d700 1 -- 172.21.15.152:0/991375438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2db00f6510 msgr2=0x7f2db0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:23.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.755+0000 7f2db484d700 1 --2- 172.21.15.152:0/991375438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2db00f6510 0x7f2db0106730 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f2da4013510 tx=0x7f2da400b040).stop 2023-12-06T03:54:23.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.755+0000 7f2db484d700 1 -- 172.21.15.152:0/991375438 shutdown_connections 2023-12-06T03:54:23.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.755+0000 7f2db484d700 1 --2- 172.21.15.152:0/991375438 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2d9802ff60 0x7f2d98032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:23.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.755+0000 7f2db484d700 1 --2- 172.21.15.152:0/991375438 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2db00f6510 0x7f2db0106730 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:23.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.755+0000 7f2db484d700 1 -- 172.21.15.152:0/991375438 >> 172.21.15.152:0/991375438 conn(0x7f2db00f1c70 msgr2=0x7f2db00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:23.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.755+0000 7f2db484d700 1 -- 172.21.15.152:0/991375438 shutdown_connections 2023-12-06T03:54:23.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:23.755+0000 7f2db484d700 1 -- 172.21.15.152:0/991375438 wait complete. 2023-12-06T03:54:23.758 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:24.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:22.393872+0000 mgr.smithi132.gokxja (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:25.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:24 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:23.754691+0000 mon.smithi132 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.152:0/991375438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:25.154 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:25.154 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:26.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:24.394320+0000 mgr.smithi132.gokxja (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:26.270 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.268+0000 7f3235acb700 1 -- 172.21.15.152:0/370689346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32300f4740 msgr2=0x7f32300f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:26.273 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.268+0000 7f3235acb700 1 --2- 172.21.15.152:0/370689346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32300f4740 0x7f32300f4b60 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f3220009fb0 tx=0x7f3220009b30).stop 2023-12-06T03:54:26.273 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.268+0000 7f3235acb700 1 -- 172.21.15.152:0/370689346 shutdown_connections 2023-12-06T03:54:26.273 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.268+0000 7f3235acb700 1 --2- 172.21.15.152:0/370689346 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32300f4740 0x7f32300f4b60 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:26.273 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.268+0000 7f3235acb700 1 -- 172.21.15.152:0/370689346 >> 172.21.15.152:0/370689346 conn(0x7f32300f0260 msgr2=0x7f32300f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:26.273 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.268+0000 7f3235acb700 1 -- 172.21.15.152:0/370689346 shutdown_connections 2023-12-06T03:54:26.273 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.268+0000 7f3235acb700 1 -- 172.21.15.152:0/370689346 wait complete. 2023-12-06T03:54:26.273 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.269+0000 7f3235acb700 1 Processor -- start 2023-12-06T03:54:26.273 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.269+0000 7f3235acb700 1 -- start start 2023-12-06T03:54:26.274 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.269+0000 7f3235acb700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32300f4740 0x7f32301066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:26.274 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.269+0000 7f3235acb700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3230106bc0 con 0x7f32300f4740 2023-12-06T03:54:26.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.269+0000 7f322f7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32300f4740 0x7f32301066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:26.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.269+0000 7f322f7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32300f4740 0x7f32301066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:39194/0 (socket says 172.21.15.152:39194) 2023-12-06T03:54:26.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.269+0000 7f322f7fe700 1 -- 172.21.15.152:0/397207644 learned_addr learned my addr 172.21.15.152:0/397207644 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:26.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.270+0000 7f322f7fe700 1 -- 172.21.15.152:0/397207644 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3220005040 con 0x7f32300f4740 2023-12-06T03:54:26.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.270+0000 7f322f7fe700 1 --2- 172.21.15.152:0/397207644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32300f4740 0x7f32301066d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f3220013510 tx=0x7f322000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:26.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.270+0000 7f3227fff700 1 -- 172.21.15.152:0/397207644 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3220009320 con 0x7f32300f4740 2023-12-06T03:54:26.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.270+0000 7f3235acb700 1 -- 172.21.15.152:0/397207644 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3230106dc0 con 0x7f32300f4740 2023-12-06T03:54:26.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.270+0000 7f3227fff700 1 -- 172.21.15.152:0/397207644 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3220013f00 con 0x7f32300f4740 2023-12-06T03:54:26.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.270+0000 7f3227fff700 1 -- 172.21.15.152:0/397207644 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3220014d90 con 0x7f32300f4740 2023-12-06T03:54:26.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.270+0000 7f3235acb700 1 -- 172.21.15.152:0/397207644 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3230107250 con 0x7f32300f4740 2023-12-06T03:54:26.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.271+0000 7f3227fff700 1 -- 172.21.15.152:0/397207644 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3220014420 con 0x7f32300f4740 2023-12-06T03:54:26.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.271+0000 7f3235acb700 1 -- 172.21.15.152:0/397207644 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3230045ad0 con 0x7f32300f4740 2023-12-06T03:54:26.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.271+0000 7f3227fff700 1 --2- 172.21.15.152:0/397207644 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f321c02ff60 0x7f321c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:26.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.271+0000 7f3227fff700 1 -- 172.21.15.152:0/397207644 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32200410e0 con 0x7f32300f4740 2023-12-06T03:54:26.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.271+0000 7f322effd700 1 --2- 172.21.15.152:0/397207644 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f321c02ff60 0x7f321c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:26.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.272+0000 7f322effd700 1 --2- 172.21.15.152:0/397207644 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f321c02ff60 0x7f321c032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f32180096a0 tx=0x7f3218006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:26.277 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.276+0000 7f3227fff700 1 -- 172.21.15.152:0/397207644 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32200095f0 con 0x7f32300f4740 2023-12-06T03:54:26.484 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.483+0000 7f3235acb700 1 -- 172.21.15.152:0/397207644 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3230108080 con 0x7f32300f4740 2023-12-06T03:54:26.485 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.483+0000 7f3227fff700 1 -- 172.21.15.152:0/397207644 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f322001db70 con 0x7f32300f4740 2023-12-06T03:54:26.485 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:26.485 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:26.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.485+0000 7f3235acb700 1 -- 172.21.15.152:0/397207644 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f321c02ff60 msgr2=0x7f321c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:26.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.485+0000 7f3235acb700 1 --2- 172.21.15.152:0/397207644 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f321c02ff60 0x7f321c032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f32180096a0 tx=0x7f3218006b40).stop 2023-12-06T03:54:26.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.485+0000 7f3235acb700 1 -- 172.21.15.152:0/397207644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32300f4740 msgr2=0x7f32301066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:26.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.485+0000 7f3235acb700 1 --2- 172.21.15.152:0/397207644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32300f4740 0x7f32301066d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f3220013510 tx=0x7f322000b040).stop 2023-12-06T03:54:26.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.485+0000 7f3235acb700 1 -- 172.21.15.152:0/397207644 shutdown_connections 2023-12-06T03:54:26.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.485+0000 7f3235acb700 1 --2- 172.21.15.152:0/397207644 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f321c02ff60 0x7f321c032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:26.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.485+0000 7f3235acb700 1 --2- 172.21.15.152:0/397207644 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f32300f4740 0x7f32301066d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:26.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.485+0000 7f3235acb700 1 -- 172.21.15.152:0/397207644 >> 172.21.15.152:0/397207644 conn(0x7f32300f0260 msgr2=0x7f32300f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:26.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.485+0000 7f3235acb700 1 -- 172.21.15.152:0/397207644 shutdown_connections 2023-12-06T03:54:26.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:26.485+0000 7f3235acb700 1 -- 172.21.15.152:0/397207644 wait complete. 2023-12-06T03:54:26.488 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:27.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:26 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:26.484725+0000 mon.smithi132 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.152:0/397207644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:27.860 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:27.860 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:28.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:26.394666+0000 mgr.smithi132.gokxja (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:28.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.955+0000 7ff04bdfa700 1 -- 172.21.15.152:0/231950043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0440f5410 msgr2=0x7ff0440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:28.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.955+0000 7ff04bdfa700 1 --2- 172.21.15.152:0/231950043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0440f5410 0x7ff0440f5830 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7ff034009fb0 tx=0x7ff034009b30).stop 2023-12-06T03:54:28.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.955+0000 7ff04bdfa700 1 -- 172.21.15.152:0/231950043 shutdown_connections 2023-12-06T03:54:28.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.955+0000 7ff04bdfa700 1 --2- 172.21.15.152:0/231950043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0440f5410 0x7ff0440f5830 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:28.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.955+0000 7ff04bdfa700 1 -- 172.21.15.152:0/231950043 >> 172.21.15.152:0/231950043 conn(0x7ff0440f0b70 msgr2=0x7ff0440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:28.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.955+0000 7ff04bdfa700 1 -- 172.21.15.152:0/231950043 shutdown_connections 2023-12-06T03:54:28.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.955+0000 7ff04bdfa700 1 -- 172.21.15.152:0/231950043 wait complete. 2023-12-06T03:54:28.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.955+0000 7ff04bdfa700 1 Processor -- start 2023-12-06T03:54:28.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.956+0000 7ff04bdfa700 1 -- start start 2023-12-06T03:54:28.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.956+0000 7ff04bdfa700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0440f5410 0x7ff04410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:28.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.956+0000 7ff04bdfa700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff044107240 con 0x7ff0440f5410 2023-12-06T03:54:28.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.956+0000 7ff049b96700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0440f5410 0x7ff04410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:28.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.956+0000 7ff049b96700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0440f5410 0x7ff04410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:39218/0 (socket says 172.21.15.152:39218) 2023-12-06T03:54:28.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.956+0000 7ff049b96700 1 -- 172.21.15.152:0/3666508025 learned_addr learned my addr 172.21.15.152:0/3666508025 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:28.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.956+0000 7ff049b96700 1 -- 172.21.15.152:0/3666508025 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff034005040 con 0x7ff0440f5410 2023-12-06T03:54:28.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.956+0000 7ff049b96700 1 --2- 172.21.15.152:0/3666508025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0440f5410 0x7ff04410b850 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff034013510 tx=0x7ff03400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:28.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.956+0000 7ff03a7fc700 1 -- 172.21.15.152:0/3666508025 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff034009320 con 0x7ff0440f5410 2023-12-06T03:54:28.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.956+0000 7ff04bdfa700 1 -- 172.21.15.152:0/3666508025 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff044060220 con 0x7ff0440f5410 2023-12-06T03:54:28.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.956+0000 7ff03a7fc700 1 -- 172.21.15.152:0/3666508025 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff034013f00 con 0x7ff0440f5410 2023-12-06T03:54:28.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.956+0000 7ff03a7fc700 1 -- 172.21.15.152:0/3666508025 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff034014d90 con 0x7ff0440f5410 2023-12-06T03:54:28.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.956+0000 7ff04bdfa700 1 -- 172.21.15.152:0/3666508025 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff04410bd40 con 0x7ff0440f5410 2023-12-06T03:54:28.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.957+0000 7ff03a7fc700 1 -- 172.21.15.152:0/3666508025 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff0340263a0 con 0x7ff0440f5410 2023-12-06T03:54:28.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.957+0000 7ff03a7fc700 1 --2- 172.21.15.152:0/3666508025 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff03002ff60 0x7ff030032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:28.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.957+0000 7ff03a7fc700 1 -- 172.21.15.152:0/3666508025 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff0340410d0 con 0x7ff0440f5410 2023-12-06T03:54:28.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.957+0000 7ff04bdfa700 1 -- 172.21.15.152:0/3666508025 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff044045ad0 con 0x7ff0440f5410 2023-12-06T03:54:28.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.958+0000 7ff049395700 1 --2- 172.21.15.152:0/3666508025 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff03002ff60 0x7ff030032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:28.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.958+0000 7ff049395700 1 --2- 172.21.15.152:0/3666508025 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff03002ff60 0x7ff030032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff0400096a0 tx=0x7ff040006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:28.964 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:28.962+0000 7ff03a7fc700 1 -- 172.21.15.152:0/3666508025 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff034012920 con 0x7ff0440f5410 2023-12-06T03:54:29.170 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:29.168+0000 7ff04bdfa700 1 -- 172.21.15.152:0/3666508025 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff04410c3c0 con 0x7ff0440f5410 2023-12-06T03:54:29.170 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:29.169+0000 7ff03a7fc700 1 -- 172.21.15.152:0/3666508025 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff03402b020 con 0x7ff0440f5410 2023-12-06T03:54:29.170 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:29.171 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:29.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:29.170+0000 7ff04bdfa700 1 -- 172.21.15.152:0/3666508025 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff03002ff60 msgr2=0x7ff030032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:29.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:29.170+0000 7ff04bdfa700 1 --2- 172.21.15.152:0/3666508025 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff03002ff60 0x7ff030032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7ff0400096a0 tx=0x7ff040006b40).stop 2023-12-06T03:54:29.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:29.170+0000 7ff04bdfa700 1 -- 172.21.15.152:0/3666508025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0440f5410 msgr2=0x7ff04410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:29.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:29.170+0000 7ff04bdfa700 1 --2- 172.21.15.152:0/3666508025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0440f5410 0x7ff04410b850 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ff034013510 tx=0x7ff03400b040).stop 2023-12-06T03:54:29.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:29.170+0000 7ff04bdfa700 1 -- 172.21.15.152:0/3666508025 shutdown_connections 2023-12-06T03:54:29.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:29.170+0000 7ff04bdfa700 1 --2- 172.21.15.152:0/3666508025 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff03002ff60 0x7ff030032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:29.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:29.170+0000 7ff04bdfa700 1 --2- 172.21.15.152:0/3666508025 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff0440f5410 0x7ff04410b850 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:29.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:29.170+0000 7ff04bdfa700 1 -- 172.21.15.152:0/3666508025 >> 172.21.15.152:0/3666508025 conn(0x7ff0440f0b70 msgr2=0x7ff0440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:29.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:29.170+0000 7ff04bdfa700 1 -- 172.21.15.152:0/3666508025 shutdown_connections 2023-12-06T03:54:29.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:29.170+0000 7ff04bdfa700 1 -- 172.21.15.152:0/3666508025 wait complete. 2023-12-06T03:54:29.173 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:30.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:28.395062+0000 mgr.smithi132.gokxja (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:30.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:29.170147+0000 mon.smithi132 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.152:0/3666508025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:30.559 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:30.559 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:31.699 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.696+0000 7fb974df2700 1 -- 172.21.15.152:0/2564716257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9700f51a0 msgr2=0x7fb9700f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:31.699 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.696+0000 7fb974df2700 1 --2- 172.21.15.152:0/2564716257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9700f51a0 0x7fb9700f55c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fb960009fb0 tx=0x7fb960009b30).stop 2023-12-06T03:54:31.699 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.697+0000 7fb974df2700 1 -- 172.21.15.152:0/2564716257 shutdown_connections 2023-12-06T03:54:31.699 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.697+0000 7fb974df2700 1 --2- 172.21.15.152:0/2564716257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9700f51a0 0x7fb9700f55c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:31.700 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.697+0000 7fb974df2700 1 -- 172.21.15.152:0/2564716257 >> 172.21.15.152:0/2564716257 conn(0x7fb9700f09a0 msgr2=0x7fb9700f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:31.700 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.697+0000 7fb974df2700 1 -- 172.21.15.152:0/2564716257 shutdown_connections 2023-12-06T03:54:31.700 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.697+0000 7fb974df2700 1 -- 172.21.15.152:0/2564716257 wait complete. 2023-12-06T03:54:31.700 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.697+0000 7fb974df2700 1 Processor -- start 2023-12-06T03:54:31.700 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.697+0000 7fb974df2700 1 -- start start 2023-12-06T03:54:31.700 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.697+0000 7fb974df2700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9700f51a0 0x7fb9700fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:31.700 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.697+0000 7fb974df2700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9700fb5a0 con 0x7fb9700f51a0 2023-12-06T03:54:31.700 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.698+0000 7fb96f7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9700f51a0 0x7fb9700fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:31.700 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.698+0000 7fb96f7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9700f51a0 0x7fb9700fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:59852/0 (socket says 172.21.15.152:59852) 2023-12-06T03:54:31.700 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.698+0000 7fb96f7fe700 1 -- 172.21.15.152:0/3097763802 learned_addr learned my addr 172.21.15.152:0/3097763802 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:31.701 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.698+0000 7fb96f7fe700 1 -- 172.21.15.152:0/3097763802 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb960005040 con 0x7fb9700f51a0 2023-12-06T03:54:31.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.698+0000 7fb96f7fe700 1 --2- 172.21.15.152:0/3097763802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9700f51a0 0x7fb9700fd120 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb960013510 tx=0x7fb96000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:31.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.698+0000 7fb957fff700 1 -- 172.21.15.152:0/3097763802 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb960009320 con 0x7fb9700f51a0 2023-12-06T03:54:31.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.698+0000 7fb974df2700 1 -- 172.21.15.152:0/3097763802 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9700fb740 con 0x7fb9700f51a0 2023-12-06T03:54:31.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.698+0000 7fb957fff700 1 -- 172.21.15.152:0/3097763802 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb960013f00 con 0x7fb9700f51a0 2023-12-06T03:54:31.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.698+0000 7fb957fff700 1 -- 172.21.15.152:0/3097763802 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb960014dc0 con 0x7fb9700f51a0 2023-12-06T03:54:31.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.698+0000 7fb974df2700 1 -- 172.21.15.152:0/3097763802 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9700fbbd0 con 0x7fb9700f51a0 2023-12-06T03:54:31.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.699+0000 7fb957fff700 1 -- 172.21.15.152:0/3097763802 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb960014420 con 0x7fb9700f51a0 2023-12-06T03:54:31.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.699+0000 7fb974df2700 1 -- 172.21.15.152:0/3097763802 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb970059100 con 0x7fb9700f51a0 2023-12-06T03:54:31.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.699+0000 7fb957fff700 1 --2- 172.21.15.152:0/3097763802 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb95802ff60 0x7fb958032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:31.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.699+0000 7fb957fff700 1 -- 172.21.15.152:0/3097763802 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb960041110 con 0x7fb9700f51a0 2023-12-06T03:54:31.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.699+0000 7fb96effd700 1 --2- 172.21.15.152:0/3097763802 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb95802ff60 0x7fb958032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:31.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.700+0000 7fb96effd700 1 --2- 172.21.15.152:0/3097763802 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb95802ff60 0x7fb958032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb9640096a0 tx=0x7fb964006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:31.705 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.704+0000 7fb957fff700 1 -- 172.21.15.152:0/3097763802 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb9600095f0 con 0x7fb9700f51a0 2023-12-06T03:54:31.914 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.912+0000 7fb974df2700 1 -- 172.21.15.152:0/3097763802 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9700fabe0 con 0x7fb9700f51a0 2023-12-06T03:54:31.915 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.913+0000 7fb957fff700 1 -- 172.21.15.152:0/3097763802 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb96001cb10 con 0x7fb9700f51a0 2023-12-06T03:54:31.915 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:31.915 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:31.916 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.914+0000 7fb974df2700 1 -- 172.21.15.152:0/3097763802 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb95802ff60 msgr2=0x7fb958032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:31.917 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.914+0000 7fb974df2700 1 --2- 172.21.15.152:0/3097763802 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb95802ff60 0x7fb958032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fb9640096a0 tx=0x7fb964006b40).stop 2023-12-06T03:54:31.917 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.914+0000 7fb974df2700 1 -- 172.21.15.152:0/3097763802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9700f51a0 msgr2=0x7fb9700fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:31.917 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.914+0000 7fb974df2700 1 --2- 172.21.15.152:0/3097763802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9700f51a0 0x7fb9700fd120 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb960013510 tx=0x7fb96000b040).stop 2023-12-06T03:54:31.917 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.914+0000 7fb974df2700 1 -- 172.21.15.152:0/3097763802 shutdown_connections 2023-12-06T03:54:31.917 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.914+0000 7fb974df2700 1 --2- 172.21.15.152:0/3097763802 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb95802ff60 0x7fb958032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:31.917 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.914+0000 7fb974df2700 1 --2- 172.21.15.152:0/3097763802 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9700f51a0 0x7fb9700fd120 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:31.917 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.914+0000 7fb974df2700 1 -- 172.21.15.152:0/3097763802 >> 172.21.15.152:0/3097763802 conn(0x7fb9700f09a0 msgr2=0x7fb9700f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:31.917 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.914+0000 7fb974df2700 1 -- 172.21.15.152:0/3097763802 shutdown_connections 2023-12-06T03:54:31.917 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:31.914+0000 7fb974df2700 1 -- 172.21.15.152:0/3097763802 wait complete. 2023-12-06T03:54:31.917 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:32.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:31 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:30.395368+0000 mgr.smithi132.gokxja (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:33.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:32 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:31.914238+0000 mon.smithi132 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.152:0/3097763802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:33.297 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:33.298 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:34.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:33 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:32.395667+0000 mgr.smithi132.gokxja (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:34.421 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.418+0000 7f17f16dd700 1 -- 172.21.15.152:0/2245308148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0f51a0 msgr2=0x7f17ec0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:34.421 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.418+0000 7f17f16dd700 1 --2- 172.21.15.152:0/2245308148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0f51a0 0x7f17ec0f55c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f17dc009fb0 tx=0x7f17dc009b30).stop 2023-12-06T03:54:34.421 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.418+0000 7f17f16dd700 1 -- 172.21.15.152:0/2245308148 shutdown_connections 2023-12-06T03:54:34.421 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.418+0000 7f17f16dd700 1 --2- 172.21.15.152:0/2245308148 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0f51a0 0x7f17ec0f55c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:34.421 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.418+0000 7f17f16dd700 1 -- 172.21.15.152:0/2245308148 >> 172.21.15.152:0/2245308148 conn(0x7f17ec0f09a0 msgr2=0x7f17ec0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:34.421 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.419+0000 7f17f16dd700 1 -- 172.21.15.152:0/2245308148 shutdown_connections 2023-12-06T03:54:34.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.419+0000 7f17f16dd700 1 -- 172.21.15.152:0/2245308148 wait complete. 2023-12-06T03:54:34.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.419+0000 7f17f16dd700 1 Processor -- start 2023-12-06T03:54:34.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.419+0000 7f17f16dd700 1 -- start start 2023-12-06T03:54:34.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.419+0000 7f17f16dd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0f51a0 0x7f17ec0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:34.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.419+0000 7f17f16dd700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ec0fb5a0 con 0x7f17ec0f51a0 2023-12-06T03:54:34.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.420+0000 7f17ebfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0f51a0 0x7f17ec0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:34.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.420+0000 7f17ebfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0f51a0 0x7f17ec0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:59876/0 (socket says 172.21.15.152:59876) 2023-12-06T03:54:34.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.420+0000 7f17ebfff700 1 -- 172.21.15.152:0/2855846986 learned_addr learned my addr 172.21.15.152:0/2855846986 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:34.423 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.420+0000 7f17ebfff700 1 -- 172.21.15.152:0/2855846986 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17dc005040 con 0x7f17ec0f51a0 2023-12-06T03:54:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.420+0000 7f17ebfff700 1 --2- 172.21.15.152:0/2855846986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0f51a0 0x7f17ec0fd120 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f17dc013510 tx=0x7f17dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.420+0000 7f17e8ff9700 1 -- 172.21.15.152:0/2855846986 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17dc009320 con 0x7f17ec0f51a0 2023-12-06T03:54:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.420+0000 7f17f16dd700 1 -- 172.21.15.152:0/2855846986 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ec0fb740 con 0x7f17ec0f51a0 2023-12-06T03:54:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.420+0000 7f17e8ff9700 1 -- 172.21.15.152:0/2855846986 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17dc013f00 con 0x7f17ec0f51a0 2023-12-06T03:54:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.420+0000 7f17e8ff9700 1 -- 172.21.15.152:0/2855846986 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17dc014dc0 con 0x7f17ec0f51a0 2023-12-06T03:54:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.420+0000 7f17f16dd700 1 -- 172.21.15.152:0/2855846986 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ec0fbbd0 con 0x7f17ec0f51a0 2023-12-06T03:54:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.421+0000 7f17e8ff9700 1 -- 172.21.15.152:0/2855846986 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f17dc0095f0 con 0x7f17ec0f51a0 2023-12-06T03:54:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.421+0000 7f17f16dd700 1 -- 172.21.15.152:0/2855846986 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ec059100 con 0x7f17ec0f51a0 2023-12-06T03:54:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.421+0000 7f17e8ff9700 1 --2- 172.21.15.152:0/2855846986 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f17d402ffb0 0x7f17d4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.421+0000 7f17e8ff9700 1 -- 172.21.15.152:0/2855846986 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17dc041050 con 0x7f17ec0f51a0 2023-12-06T03:54:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.421+0000 7f17eb7fe700 1 --2- 172.21.15.152:0/2855846986 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f17d402ffb0 0x7f17d4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.422+0000 7f17eb7fe700 1 --2- 172.21.15.152:0/2855846986 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f17d402ffb0 0x7f17d4032470 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f17e00096a0 tx=0x7f17e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:34.427 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.426+0000 7f17e8ff9700 1 -- 172.21.15.152:0/2855846986 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17dc014420 con 0x7f17ec0f51a0 2023-12-06T03:54:34.635 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.633+0000 7f17f16dd700 1 -- 172.21.15.152:0/2855846986 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17ec0fabe0 con 0x7f17ec0f51a0 2023-12-06T03:54:34.635 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.634+0000 7f17e8ff9700 1 -- 172.21.15.152:0/2855846986 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f17dc010230 con 0x7f17ec0f51a0 2023-12-06T03:54:34.636 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:34.636 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:34.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.636+0000 7f17f16dd700 1 -- 172.21.15.152:0/2855846986 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f17d402ffb0 msgr2=0x7f17d4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:34.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.636+0000 7f17f16dd700 1 --2- 172.21.15.152:0/2855846986 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f17d402ffb0 0x7f17d4032470 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f17e00096a0 tx=0x7f17e0006b40).stop 2023-12-06T03:54:34.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.636+0000 7f17f16dd700 1 -- 172.21.15.152:0/2855846986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0f51a0 msgr2=0x7f17ec0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:34.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.636+0000 7f17f16dd700 1 --2- 172.21.15.152:0/2855846986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0f51a0 0x7f17ec0fd120 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f17dc013510 tx=0x7f17dc00b040).stop 2023-12-06T03:54:34.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.636+0000 7f17f16dd700 1 -- 172.21.15.152:0/2855846986 shutdown_connections 2023-12-06T03:54:34.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.636+0000 7f17f16dd700 1 --2- 172.21.15.152:0/2855846986 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f17d402ffb0 0x7f17d4032470 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:34.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.636+0000 7f17f16dd700 1 --2- 172.21.15.152:0/2855846986 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f17ec0f51a0 0x7f17ec0fd120 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:34.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.636+0000 7f17f16dd700 1 -- 172.21.15.152:0/2855846986 >> 172.21.15.152:0/2855846986 conn(0x7f17ec0f09a0 msgr2=0x7f17ec05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:34.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.636+0000 7f17f16dd700 1 -- 172.21.15.152:0/2855846986 shutdown_connections 2023-12-06T03:54:34.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:34.636+0000 7f17f16dd700 1 -- 172.21.15.152:0/2855846986 wait complete. 2023-12-06T03:54:34.639 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:35.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:34 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:34.635405+0000 mon.smithi132 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.152:0/2855846986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:36.030 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:36.030 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:36.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:35 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:34.396060+0000 mgr.smithi132.gokxja (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:37.168 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.162+0000 7fc43e163700 1 -- 172.21.15.152:0/582516084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4380f5410 msgr2=0x7fc4380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:37.168 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.162+0000 7fc43e163700 1 --2- 172.21.15.152:0/582516084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4380f5410 0x7fc4380f5830 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fc428009fb0 tx=0x7fc428009b30).stop 2023-12-06T03:54:37.168 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.162+0000 7fc43e163700 1 -- 172.21.15.152:0/582516084 shutdown_connections 2023-12-06T03:54:37.168 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.162+0000 7fc43e163700 1 --2- 172.21.15.152:0/582516084 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4380f5410 0x7fc4380f5830 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:37.168 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.162+0000 7fc43e163700 1 -- 172.21.15.152:0/582516084 >> 172.21.15.152:0/582516084 conn(0x7fc4380f0b70 msgr2=0x7fc4380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:37.168 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.162+0000 7fc43e163700 1 -- 172.21.15.152:0/582516084 shutdown_connections 2023-12-06T03:54:37.168 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.162+0000 7fc43e163700 1 -- 172.21.15.152:0/582516084 wait complete. 2023-12-06T03:54:37.168 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.163+0000 7fc43e163700 1 Processor -- start 2023-12-06T03:54:37.168 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.163+0000 7fc43e163700 1 -- start start 2023-12-06T03:54:37.168 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.163+0000 7fc43e163700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4380f5410 0x7fc438109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:37.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.163+0000 7fc43e163700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc438060240 con 0x7fc4380f5410 2023-12-06T03:54:37.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.163+0000 7fc4377fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4380f5410 0x7fc438109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:37.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.163+0000 7fc4377fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4380f5410 0x7fc438109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:59898/0 (socket says 172.21.15.152:59898) 2023-12-06T03:54:37.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.163+0000 7fc4377fe700 1 -- 172.21.15.152:0/3867568896 learned_addr learned my addr 172.21.15.152:0/3867568896 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:37.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.164+0000 7fc4377fe700 1 -- 172.21.15.152:0/3867568896 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc428005040 con 0x7fc4380f5410 2023-12-06T03:54:37.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.164+0000 7fc4377fe700 1 --2- 172.21.15.152:0/3867568896 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4380f5410 0x7fc438109430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fc428014000 tx=0x7fc42800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:37.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.164+0000 7fc42ffff700 1 -- 172.21.15.152:0/3867568896 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc4280095f0 con 0x7fc4380f5410 2023-12-06T03:54:37.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.164+0000 7fc43e163700 1 -- 172.21.15.152:0/3867568896 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc438060440 con 0x7fc4380f5410 2023-12-06T03:54:37.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.164+0000 7fc42ffff700 1 -- 172.21.15.152:0/3867568896 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc428007e70 con 0x7fc4380f5410 2023-12-06T03:54:37.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.164+0000 7fc42ffff700 1 -- 172.21.15.152:0/3867568896 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc428014ea0 con 0x7fc4380f5410 2023-12-06T03:54:37.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.164+0000 7fc43e163700 1 -- 172.21.15.152:0/3867568896 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc438109ae0 con 0x7fc4380f5410 2023-12-06T03:54:37.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.165+0000 7fc42ffff700 1 -- 172.21.15.152:0/3867568896 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc428025070 con 0x7fc4380f5410 2023-12-06T03:54:37.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.165+0000 7fc43e163700 1 -- 172.21.15.152:0/3867568896 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc438045ad0 con 0x7fc4380f5410 2023-12-06T03:54:37.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.165+0000 7fc42ffff700 1 --2- 172.21.15.152:0/3867568896 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc42402ff60 0x7fc424032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:37.170 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.165+0000 7fc42ffff700 1 -- 172.21.15.152:0/3867568896 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4280409e0 con 0x7fc4380f5410 2023-12-06T03:54:37.170 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.165+0000 7fc436ffd700 1 --2- 172.21.15.152:0/3867568896 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc42402ff60 0x7fc424032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:37.170 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.166+0000 7fc436ffd700 1 --2- 172.21.15.152:0/3867568896 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc42402ff60 0x7fc424032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc4200096a0 tx=0x7fc420006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:37.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.170+0000 7fc42ffff700 1 -- 172.21.15.152:0/3867568896 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc42800f4e0 con 0x7fc4380f5410 2023-12-06T03:54:37.380 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.378+0000 7fc43e163700 1 -- 172.21.15.152:0/3867568896 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4380fae60 con 0x7fc4380f5410 2023-12-06T03:54:37.381 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.379+0000 7fc42ffff700 1 -- 172.21.15.152:0/3867568896 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc42800f300 con 0x7fc4380f5410 2023-12-06T03:54:37.381 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:37.381 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:37.382 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.381+0000 7fc43e163700 1 -- 172.21.15.152:0/3867568896 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc42402ff60 msgr2=0x7fc424032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:37.383 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.381+0000 7fc43e163700 1 --2- 172.21.15.152:0/3867568896 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc42402ff60 0x7fc424032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fc4200096a0 tx=0x7fc420006b40).stop 2023-12-06T03:54:37.383 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.381+0000 7fc43e163700 1 -- 172.21.15.152:0/3867568896 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4380f5410 msgr2=0x7fc438109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:37.383 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.381+0000 7fc43e163700 1 --2- 172.21.15.152:0/3867568896 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4380f5410 0x7fc438109430 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fc428014000 tx=0x7fc42800b040).stop 2023-12-06T03:54:37.383 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.381+0000 7fc43e163700 1 -- 172.21.15.152:0/3867568896 shutdown_connections 2023-12-06T03:54:37.383 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.381+0000 7fc43e163700 1 --2- 172.21.15.152:0/3867568896 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc42402ff60 0x7fc424032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:37.383 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.381+0000 7fc43e163700 1 --2- 172.21.15.152:0/3867568896 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4380f5410 0x7fc438109430 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:37.383 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.381+0000 7fc43e163700 1 -- 172.21.15.152:0/3867568896 >> 172.21.15.152:0/3867568896 conn(0x7fc4380f0b70 msgr2=0x7fc4380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:37.383 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.381+0000 7fc43e163700 1 -- 172.21.15.152:0/3867568896 shutdown_connections 2023-12-06T03:54:37.383 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:37.381+0000 7fc43e163700 1 -- 172.21.15.152:0/3867568896 wait complete. 2023-12-06T03:54:37.383 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:38.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:36.396403+0000 mgr.smithi132.gokxja (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:38.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:37.380579+0000 mon.smithi132 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.152:0/3867568896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:38.763 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:38.763 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:39.936 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.933+0000 7fb3370c3700 1 -- 172.21.15.152:0/4205991700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3300f51a0 msgr2=0x7fb3300f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:39.936 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.933+0000 7fb3370c3700 1 --2- 172.21.15.152:0/4205991700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3300f51a0 0x7fb3300f55c0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7fb320009fb0 tx=0x7fb320009b30).stop 2023-12-06T03:54:39.936 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.933+0000 7fb3370c3700 1 -- 172.21.15.152:0/4205991700 shutdown_connections 2023-12-06T03:54:39.936 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.933+0000 7fb3370c3700 1 --2- 172.21.15.152:0/4205991700 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3300f51a0 0x7fb3300f55c0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:39.936 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.933+0000 7fb3370c3700 1 -- 172.21.15.152:0/4205991700 >> 172.21.15.152:0/4205991700 conn(0x7fb3300f09a0 msgr2=0x7fb3300f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:39.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.933+0000 7fb3370c3700 1 -- 172.21.15.152:0/4205991700 shutdown_connections 2023-12-06T03:54:39.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.933+0000 7fb3370c3700 1 -- 172.21.15.152:0/4205991700 wait complete. 2023-12-06T03:54:39.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.933+0000 7fb3370c3700 1 Processor -- start 2023-12-06T03:54:39.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.933+0000 7fb3370c3700 1 -- start start 2023-12-06T03:54:39.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.934+0000 7fb3370c3700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3300f51a0 0x7fb3300fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:39.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.934+0000 7fb3370c3700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3300fb5a0 con 0x7fb3300f51a0 2023-12-06T03:54:39.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.934+0000 7fb3360c1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3300f51a0 0x7fb3300fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:39.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.934+0000 7fb3360c1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3300f51a0 0x7fb3300fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40824/0 (socket says 172.21.15.152:40824) 2023-12-06T03:54:39.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.934+0000 7fb3360c1700 1 -- 172.21.15.152:0/3990330762 learned_addr learned my addr 172.21.15.152:0/3990330762 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:39.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.934+0000 7fb3360c1700 1 -- 172.21.15.152:0/3990330762 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb320005040 con 0x7fb3300f51a0 2023-12-06T03:54:39.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.934+0000 7fb3360c1700 1 --2- 172.21.15.152:0/3990330762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3300f51a0 0x7fb3300fd120 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb320009830 tx=0x7fb32000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:39.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.934+0000 7fb326ffd700 1 -- 172.21.15.152:0/3990330762 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb320009320 con 0x7fb3300f51a0 2023-12-06T03:54:39.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.934+0000 7fb3370c3700 1 -- 172.21.15.152:0/3990330762 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3300fb740 con 0x7fb3300f51a0 2023-12-06T03:54:39.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.934+0000 7fb326ffd700 1 -- 172.21.15.152:0/3990330762 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3200136c0 con 0x7fb3300f51a0 2023-12-06T03:54:39.938 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.934+0000 7fb3370c3700 1 -- 172.21.15.152:0/3990330762 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3300fbbd0 con 0x7fb3300f51a0 2023-12-06T03:54:39.938 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.934+0000 7fb326ffd700 1 -- 172.21.15.152:0/3990330762 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb32001ee90 con 0x7fb3300f51a0 2023-12-06T03:54:39.939 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.935+0000 7fb326ffd700 1 -- 172.21.15.152:0/3990330762 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb32001e420 con 0x7fb3300f51a0 2023-12-06T03:54:39.939 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.935+0000 7fb3370c3700 1 -- 172.21.15.152:0/3990330762 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb330059100 con 0x7fb3300f51a0 2023-12-06T03:54:39.939 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.935+0000 7fb326ffd700 1 --2- 172.21.15.152:0/3990330762 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb31c02ff10 0x7fb31c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:39.939 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.935+0000 7fb326ffd700 1 -- 172.21.15.152:0/3990330762 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3200410d0 con 0x7fb3300f51a0 2023-12-06T03:54:39.939 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.936+0000 7fb3358c0700 1 --2- 172.21.15.152:0/3990330762 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb31c02ff10 0x7fb31c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:39.939 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.936+0000 7fb3358c0700 1 --2- 172.21.15.152:0/3990330762 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb31c02ff10 0x7fb31c0323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb32c0096a0 tx=0x7fb32c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:39.942 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:39.941+0000 7fb326ffd700 1 -- 172.21.15.152:0/3990330762 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3200095f0 con 0x7fb3300f51a0 2023-12-06T03:54:40.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:39 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:38.396770+0000 mgr.smithi132.gokxja (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:40.146 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:40.145+0000 7fb3370c3700 1 -- 172.21.15.152:0/3990330762 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3300fabe0 con 0x7fb3300f51a0 2023-12-06T03:54:40.147 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:40.145+0000 7fb326ffd700 1 -- 172.21.15.152:0/3990330762 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb32002c2d0 con 0x7fb3300f51a0 2023-12-06T03:54:40.147 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:40.148 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:40.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:40.147+0000 7fb3370c3700 1 -- 172.21.15.152:0/3990330762 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb31c02ff10 msgr2=0x7fb31c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:40.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:40.147+0000 7fb3370c3700 1 --2- 172.21.15.152:0/3990330762 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb31c02ff10 0x7fb31c0323d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fb32c0096a0 tx=0x7fb32c006b40).stop 2023-12-06T03:54:40.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:40.147+0000 7fb3370c3700 1 -- 172.21.15.152:0/3990330762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3300f51a0 msgr2=0x7fb3300fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:40.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:40.147+0000 7fb3370c3700 1 --2- 172.21.15.152:0/3990330762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3300f51a0 0x7fb3300fd120 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fb320009830 tx=0x7fb32000b040).stop 2023-12-06T03:54:40.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:40.147+0000 7fb3370c3700 1 -- 172.21.15.152:0/3990330762 shutdown_connections 2023-12-06T03:54:40.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:40.147+0000 7fb3370c3700 1 --2- 172.21.15.152:0/3990330762 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb31c02ff10 0x7fb31c0323d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:40.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:40.147+0000 7fb3370c3700 1 --2- 172.21.15.152:0/3990330762 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb3300f51a0 0x7fb3300fd120 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:40.150 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:40.147+0000 7fb3370c3700 1 -- 172.21.15.152:0/3990330762 >> 172.21.15.152:0/3990330762 conn(0x7fb3300f09a0 msgr2=0x7fb33005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:40.150 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:40.147+0000 7fb3370c3700 1 -- 172.21.15.152:0/3990330762 shutdown_connections 2023-12-06T03:54:40.150 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:40.147+0000 7fb3370c3700 1 -- 172.21.15.152:0/3990330762 wait complete. 2023-12-06T03:54:40.150 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:41.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:40 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:40.146786+0000 mon.smithi132 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.152:0/3990330762' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:41.478 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:41.479 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:42.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:41 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:40.397180+0000 mgr.smithi132.gokxja (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:42.544 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.542+0000 7fa10f449700 1 -- 172.21.15.152:0/3094094262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1080f62a0 msgr2=0x7fa1080f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:42.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.542+0000 7fa10f449700 1 --2- 172.21.15.152:0/3094094262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1080f62a0 0x7fa1080f66c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7fa104009fb0 tx=0x7fa104009b30).stop 2023-12-06T03:54:42.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.542+0000 7fa10f449700 1 -- 172.21.15.152:0/3094094262 shutdown_connections 2023-12-06T03:54:42.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.542+0000 7fa10f449700 1 --2- 172.21.15.152:0/3094094262 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1080f62a0 0x7fa1080f66c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:42.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.542+0000 7fa10f449700 1 -- 172.21.15.152:0/3094094262 >> 172.21.15.152:0/3094094262 conn(0x7fa1080f1aa0 msgr2=0x7fa1080f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:42.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.542+0000 7fa10f449700 1 -- 172.21.15.152:0/3094094262 shutdown_connections 2023-12-06T03:54:42.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.542+0000 7fa10f449700 1 -- 172.21.15.152:0/3094094262 wait complete. 2023-12-06T03:54:42.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.542+0000 7fa10f449700 1 Processor -- start 2023-12-06T03:54:42.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.543+0000 7fa10f449700 1 -- start start 2023-12-06T03:54:42.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.543+0000 7fa10f449700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1080f62a0 0x7fa1081086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:42.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.543+0000 7fa10f449700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa108108ba0 con 0x7fa1080f62a0 2023-12-06T03:54:42.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.543+0000 7fa10e447700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1080f62a0 0x7fa1081086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:42.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.543+0000 7fa10e447700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1080f62a0 0x7fa1081086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40836/0 (socket says 172.21.15.152:40836) 2023-12-06T03:54:42.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.543+0000 7fa10e447700 1 -- 172.21.15.152:0/1397213848 learned_addr learned my addr 172.21.15.152:0/1397213848 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:42.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.543+0000 7fa10e447700 1 -- 172.21.15.152:0/1397213848 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa104005040 con 0x7fa1080f62a0 2023-12-06T03:54:42.546 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.543+0000 7fa10e447700 1 --2- 172.21.15.152:0/1397213848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1080f62a0 0x7fa1081086b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa104013510 tx=0x7fa10400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:42.546 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.543+0000 7fa0feffd700 1 -- 172.21.15.152:0/1397213848 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa104009320 con 0x7fa1080f62a0 2023-12-06T03:54:42.546 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.543+0000 7fa10f449700 1 -- 172.21.15.152:0/1397213848 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa108108da0 con 0x7fa1080f62a0 2023-12-06T03:54:42.546 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.543+0000 7fa0feffd700 1 -- 172.21.15.152:0/1397213848 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa104013f00 con 0x7fa1080f62a0 2023-12-06T03:54:42.546 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.543+0000 7fa0feffd700 1 -- 172.21.15.152:0/1397213848 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa104014dc0 con 0x7fa1080f62a0 2023-12-06T03:54:42.546 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.544+0000 7fa10f449700 1 -- 172.21.15.152:0/1397213848 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa108109230 con 0x7fa1080f62a0 2023-12-06T03:54:42.547 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.544+0000 7fa0feffd700 1 -- 172.21.15.152:0/1397213848 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa10401d3d0 con 0x7fa1080f62a0 2023-12-06T03:54:42.547 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.544+0000 7fa0feffd700 1 --2- 172.21.15.152:0/1397213848 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa0f402ff60 0x7fa0f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:42.547 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.544+0000 7fa10f449700 1 -- 172.21.15.152:0/1397213848 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa108045ad0 con 0x7fa1080f62a0 2023-12-06T03:54:42.547 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.544+0000 7fa0feffd700 1 -- 172.21.15.152:0/1397213848 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1040413b0 con 0x7fa1080f62a0 2023-12-06T03:54:42.547 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.545+0000 7fa10dc46700 1 --2- 172.21.15.152:0/1397213848 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa0f402ff60 0x7fa0f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:42.547 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.545+0000 7fa10dc46700 1 --2- 172.21.15.152:0/1397213848 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa0f402ff60 0x7fa0f4032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa0f80096a0 tx=0x7fa0f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:42.550 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.549+0000 7fa0feffd700 1 -- 172.21.15.152:0/1397213848 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa104012900 con 0x7fa1080f62a0 2023-12-06T03:54:42.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.757+0000 7fa10f449700 1 -- 172.21.15.152:0/1397213848 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa10810a0e0 con 0x7fa1080f62a0 2023-12-06T03:54:42.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.758+0000 7fa0feffd700 1 -- 172.21.15.152:0/1397213848 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa10402b030 con 0x7fa1080f62a0 2023-12-06T03:54:42.758 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:42.758 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:42.760 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.759+0000 7fa10f449700 1 -- 172.21.15.152:0/1397213848 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa0f402ff60 msgr2=0x7fa0f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:42.760 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.759+0000 7fa10f449700 1 --2- 172.21.15.152:0/1397213848 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa0f402ff60 0x7fa0f4032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa0f80096a0 tx=0x7fa0f8006b40).stop 2023-12-06T03:54:42.760 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.759+0000 7fa10f449700 1 -- 172.21.15.152:0/1397213848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1080f62a0 msgr2=0x7fa1081086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:42.760 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.759+0000 7fa10f449700 1 --2- 172.21.15.152:0/1397213848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1080f62a0 0x7fa1081086b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7fa104013510 tx=0x7fa10400b040).stop 2023-12-06T03:54:42.760 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.759+0000 7fa10f449700 1 -- 172.21.15.152:0/1397213848 shutdown_connections 2023-12-06T03:54:42.760 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.759+0000 7fa10f449700 1 --2- 172.21.15.152:0/1397213848 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa0f402ff60 0x7fa0f4032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:42.760 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.759+0000 7fa10f449700 1 --2- 172.21.15.152:0/1397213848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa1080f62a0 0x7fa1081086b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:42.760 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.759+0000 7fa10f449700 1 -- 172.21.15.152:0/1397213848 >> 172.21.15.152:0/1397213848 conn(0x7fa1080f1aa0 msgr2=0x7fa1080f2490 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:42.760 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.759+0000 7fa10f449700 1 -- 172.21.15.152:0/1397213848 shutdown_connections 2023-12-06T03:54:42.761 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:42.759+0000 7fa10f449700 1 -- 172.21.15.152:0/1397213848 wait complete. 2023-12-06T03:54:42.761 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:44.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:42.397490+0000 mgr.smithi132.gokxja (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:44.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:42.757977+0000 mon.smithi132 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.152:0/1397213848' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:44.116 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:44.116 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:45.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.273+0000 7f00657f9700 1 -- 172.21.15.152:0/363774207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00600f47c0 msgr2=0x7f00600f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:45.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.273+0000 7f00657f9700 1 --2- 172.21.15.152:0/363774207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00600f47c0 0x7f00600f4be0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f0048009fb0 tx=0x7f0048009b30).stop 2023-12-06T03:54:45.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.273+0000 7f00657f9700 1 -- 172.21.15.152:0/363774207 shutdown_connections 2023-12-06T03:54:45.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.273+0000 7f00657f9700 1 --2- 172.21.15.152:0/363774207 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00600f47c0 0x7f00600f4be0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:45.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.273+0000 7f00657f9700 1 -- 172.21.15.152:0/363774207 >> 172.21.15.152:0/363774207 conn(0x7f00600f02e0 msgr2=0x7f00600f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:45.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.273+0000 7f00657f9700 1 -- 172.21.15.152:0/363774207 shutdown_connections 2023-12-06T03:54:45.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.273+0000 7f00657f9700 1 -- 172.21.15.152:0/363774207 wait complete. 2023-12-06T03:54:45.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.274+0000 7f00657f9700 1 Processor -- start 2023-12-06T03:54:45.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.274+0000 7f00657f9700 1 -- start start 2023-12-06T03:54:45.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.274+0000 7f00657f9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00600f47c0 0x7f0060106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:45.275 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.274+0000 7f00657f9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0060106bf0 con 0x7f00600f47c0 2023-12-06T03:54:45.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.274+0000 7f005effd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00600f47c0 0x7f0060106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:45.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.274+0000 7f005effd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00600f47c0 0x7f0060106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40856/0 (socket says 172.21.15.152:40856) 2023-12-06T03:54:45.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.274+0000 7f005effd700 1 -- 172.21.15.152:0/281046325 learned_addr learned my addr 172.21.15.152:0/281046325 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:45.278 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.274+0000 7f005effd700 1 -- 172.21.15.152:0/281046325 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0048005040 con 0x7f00600f47c0 2023-12-06T03:54:45.278 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.274+0000 7f005effd700 1 --2- 172.21.15.152:0/281046325 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00600f47c0 0x7f0060106700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f0048013510 tx=0x7f004800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:45.278 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.275+0000 7f00577fe700 1 -- 172.21.15.152:0/281046325 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0048009320 con 0x7f00600f47c0 2023-12-06T03:54:45.278 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.275+0000 7f00577fe700 1 -- 172.21.15.152:0/281046325 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0048013f00 con 0x7f00600f47c0 2023-12-06T03:54:45.278 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.275+0000 7f00577fe700 1 -- 172.21.15.152:0/281046325 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0048014d90 con 0x7f00600f47c0 2023-12-06T03:54:45.278 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.275+0000 7f00657f9700 1 -- 172.21.15.152:0/281046325 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0060106df0 con 0x7f00600f47c0 2023-12-06T03:54:45.278 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.275+0000 7f00657f9700 1 -- 172.21.15.152:0/281046325 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0060107280 con 0x7f00600f47c0 2023-12-06T03:54:45.278 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.275+0000 7f00577fe700 1 -- 172.21.15.152:0/281046325 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f00480263a0 con 0x7f00600f47c0 2023-12-06T03:54:45.278 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.276+0000 7f00577fe700 1 --2- 172.21.15.152:0/281046325 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f004c02ffb0 0x7f004c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:45.279 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.276+0000 7f00657f9700 1 -- 172.21.15.152:0/281046325 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0060045ad0 con 0x7f00600f47c0 2023-12-06T03:54:45.279 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.276+0000 7f00577fe700 1 -- 172.21.15.152:0/281046325 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0048041190 con 0x7f00600f47c0 2023-12-06T03:54:45.279 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.276+0000 7f005e7fc700 1 --2- 172.21.15.152:0/281046325 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f004c02ffb0 0x7f004c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:45.279 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.276+0000 7f005e7fc700 1 --2- 172.21.15.152:0/281046325 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f004c02ffb0 0x7f004c032470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f00500096a0 tx=0x7f0050006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:45.281 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.280+0000 7f00577fe700 1 -- 172.21.15.152:0/281046325 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f004801ccf0 con 0x7f00600f47c0 2023-12-06T03:54:45.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.486+0000 7f00657f9700 1 -- 172.21.15.152:0/281046325 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0060108130 con 0x7f00600f47c0 2023-12-06T03:54:45.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.486+0000 7f00577fe700 1 -- 172.21.15.152:0/281046325 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f004801cb10 con 0x7f00600f47c0 2023-12-06T03:54:45.487 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:45.487 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:45.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.488+0000 7f00657f9700 1 -- 172.21.15.152:0/281046325 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f004c02ffb0 msgr2=0x7f004c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:45.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.488+0000 7f00657f9700 1 --2- 172.21.15.152:0/281046325 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f004c02ffb0 0x7f004c032470 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f00500096a0 tx=0x7f0050006b40).stop 2023-12-06T03:54:45.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.488+0000 7f00657f9700 1 -- 172.21.15.152:0/281046325 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00600f47c0 msgr2=0x7f0060106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:45.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.488+0000 7f00657f9700 1 --2- 172.21.15.152:0/281046325 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00600f47c0 0x7f0060106700 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f0048013510 tx=0x7f004800b040).stop 2023-12-06T03:54:45.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.488+0000 7f00657f9700 1 -- 172.21.15.152:0/281046325 shutdown_connections 2023-12-06T03:54:45.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.488+0000 7f00657f9700 1 --2- 172.21.15.152:0/281046325 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f004c02ffb0 0x7f004c032470 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:45.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.488+0000 7f00657f9700 1 --2- 172.21.15.152:0/281046325 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f00600f47c0 0x7f0060106700 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:45.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.488+0000 7f00657f9700 1 -- 172.21.15.152:0/281046325 >> 172.21.15.152:0/281046325 conn(0x7f00600f02e0 msgr2=0x7f00600f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:45.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.488+0000 7f00657f9700 1 -- 172.21.15.152:0/281046325 shutdown_connections 2023-12-06T03:54:45.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:45.488+0000 7f00657f9700 1 -- 172.21.15.152:0/281046325 wait complete. 2023-12-06T03:54:45.489 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:46.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:45 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:44.397802+0000 mgr.smithi132.gokxja (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:46.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:45 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:45.486689+0000 mon.smithi132 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.152:0/281046325' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:46.869 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:46.869 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:47.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.966+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/687475109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4f40f5410 msgr2=0x7fa4f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:47.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.966+0000 7fa4fb8b0700 1 --2- 172.21.15.152:0/687475109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4f40f5410 0x7fa4f40f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fa4e8009fb0 tx=0x7fa4e8009b30).stop 2023-12-06T03:54:47.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.966+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/687475109 shutdown_connections 2023-12-06T03:54:47.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.966+0000 7fa4fb8b0700 1 --2- 172.21.15.152:0/687475109 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4f40f5410 0x7fa4f40f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:47.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.966+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/687475109 >> 172.21.15.152:0/687475109 conn(0x7fa4f40f0b70 msgr2=0x7fa4f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:47.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.966+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/687475109 shutdown_connections 2023-12-06T03:54:47.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.966+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/687475109 wait complete. 2023-12-06T03:54:47.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.966+0000 7fa4fb8b0700 1 Processor -- start 2023-12-06T03:54:47.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.967+0000 7fa4fb8b0700 1 -- start start 2023-12-06T03:54:47.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.967+0000 7fa4fb8b0700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4f40f5410 0x7fa4f4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:47.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.967+0000 7fa4fb8b0700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4f4060240 con 0x7fa4f40f5410 2023-12-06T03:54:47.969 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.967+0000 7fa4f964c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4f40f5410 0x7fa4f4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:47.969 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.967+0000 7fa4f964c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4f40f5410 0x7fa4f4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40868/0 (socket says 172.21.15.152:40868) 2023-12-06T03:54:47.969 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.967+0000 7fa4f964c700 1 -- 172.21.15.152:0/3561679113 learned_addr learned my addr 172.21.15.152:0/3561679113 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:47.969 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.967+0000 7fa4f964c700 1 -- 172.21.15.152:0/3561679113 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4e8005040 con 0x7fa4f40f5410 2023-12-06T03:54:47.969 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.967+0000 7fa4f964c700 1 --2- 172.21.15.152:0/3561679113 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4f40f5410 0x7fa4f4109430 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa4e8013510 tx=0x7fa4e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:47.970 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.967+0000 7fa4e5ffb700 1 -- 172.21.15.152:0/3561679113 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4e8009320 con 0x7fa4f40f5410 2023-12-06T03:54:47.970 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.967+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/3561679113 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4f4060440 con 0x7fa4f40f5410 2023-12-06T03:54:47.970 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.967+0000 7fa4e5ffb700 1 -- 172.21.15.152:0/3561679113 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4e8013f00 con 0x7fa4f40f5410 2023-12-06T03:54:47.970 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.967+0000 7fa4e5ffb700 1 -- 172.21.15.152:0/3561679113 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4e8014d90 con 0x7fa4f40f5410 2023-12-06T03:54:47.970 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.968+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/3561679113 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4f4109ae0 con 0x7fa4f40f5410 2023-12-06T03:54:47.970 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.968+0000 7fa4e5ffb700 1 -- 172.21.15.152:0/3561679113 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4e80095f0 con 0x7fa4f40f5410 2023-12-06T03:54:47.970 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.968+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/3561679113 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4f4045ad0 con 0x7fa4f40f5410 2023-12-06T03:54:47.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.969+0000 7fa4e5ffb700 1 --2- 172.21.15.152:0/3561679113 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa4e002ff60 0x7fa4e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:47.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.969+0000 7fa4e5ffb700 1 -- 172.21.15.152:0/3561679113 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa4e80410a0 con 0x7fa4f40f5410 2023-12-06T03:54:47.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.969+0000 7fa4f8e4b700 1 --2- 172.21.15.152:0/3561679113 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa4e002ff60 0x7fa4e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:47.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.969+0000 7fa4f8e4b700 1 --2- 172.21.15.152:0/3561679113 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa4e002ff60 0x7fa4e0032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fa4f00096a0 tx=0x7fa4f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:47.974 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:47.973+0000 7fa4e5ffb700 1 -- 172.21.15.152:0/3561679113 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa4e8014420 con 0x7fa4f40f5410 2023-12-06T03:54:48.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:47 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:46.398180+0000 mgr.smithi132.gokxja (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:48.181 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:48.180+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/3561679113 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa4f40fae60 con 0x7fa4f40f5410 2023-12-06T03:54:48.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:48.181+0000 7fa4e5ffb700 1 -- 172.21.15.152:0/3561679113 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa4e8010230 con 0x7fa4f40f5410 2023-12-06T03:54:48.183 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:48.183 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:48.184 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:48.183+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/3561679113 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa4e002ff60 msgr2=0x7fa4e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:48.184 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:48.183+0000 7fa4fb8b0700 1 --2- 172.21.15.152:0/3561679113 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa4e002ff60 0x7fa4e0032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fa4f00096a0 tx=0x7fa4f0006b40).stop 2023-12-06T03:54:48.184 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:48.183+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/3561679113 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4f40f5410 msgr2=0x7fa4f4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:48.184 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:48.183+0000 7fa4fb8b0700 1 --2- 172.21.15.152:0/3561679113 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4f40f5410 0x7fa4f4109430 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa4e8013510 tx=0x7fa4e800b040).stop 2023-12-06T03:54:48.184 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:48.183+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/3561679113 shutdown_connections 2023-12-06T03:54:48.184 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:48.183+0000 7fa4fb8b0700 1 --2- 172.21.15.152:0/3561679113 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa4e002ff60 0x7fa4e0032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:48.184 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:48.183+0000 7fa4fb8b0700 1 --2- 172.21.15.152:0/3561679113 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa4f40f5410 0x7fa4f4109430 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:48.184 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:48.183+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/3561679113 >> 172.21.15.152:0/3561679113 conn(0x7fa4f40f0b70 msgr2=0x7fa4f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:48.184 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:48.183+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/3561679113 shutdown_connections 2023-12-06T03:54:48.184 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:48.183+0000 7fa4fb8b0700 1 -- 172.21.15.152:0/3561679113 wait complete. 2023-12-06T03:54:48.185 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:49.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:48 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:48.181642+0000 mon.smithi132 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.152:0/3561679113' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:49.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:49.550 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:50.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:49 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:48.398503+0000 mgr.smithi132.gokxja (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:50.695 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.693+0000 7f5242dd9700 1 -- 172.21.15.152:0/767952270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f523c0f62a0 msgr2=0x7f523c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:50.695 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.693+0000 7f5242dd9700 1 --2- 172.21.15.152:0/767952270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f523c0f62a0 0x7f523c0f66c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f522c009fb0 tx=0x7f522c009b30).stop 2023-12-06T03:54:50.695 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.693+0000 7f5242dd9700 1 -- 172.21.15.152:0/767952270 shutdown_connections 2023-12-06T03:54:50.695 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.693+0000 7f5242dd9700 1 --2- 172.21.15.152:0/767952270 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f523c0f62a0 0x7f523c0f66c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:50.695 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.693+0000 7f5242dd9700 1 -- 172.21.15.152:0/767952270 >> 172.21.15.152:0/767952270 conn(0x7f523c0f1aa0 msgr2=0x7f523c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:50.695 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.693+0000 7f5242dd9700 1 -- 172.21.15.152:0/767952270 shutdown_connections 2023-12-06T03:54:50.695 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.693+0000 7f5242dd9700 1 -- 172.21.15.152:0/767952270 wait complete. 2023-12-06T03:54:50.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.693+0000 7f5242dd9700 1 Processor -- start 2023-12-06T03:54:50.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.693+0000 7f5242dd9700 1 -- start start 2023-12-06T03:54:50.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.694+0000 7f5242dd9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f523c0f62a0 0x7f523c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:50.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.694+0000 7f5242dd9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f523c1069a0 con 0x7f523c0f62a0 2023-12-06T03:54:50.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.694+0000 7f5241dd7700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f523c0f62a0 0x7f523c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:50.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.694+0000 7f5241dd7700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f523c0f62a0 0x7f523c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:46230/0 (socket says 172.21.15.152:46230) 2023-12-06T03:54:50.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.694+0000 7f5241dd7700 1 -- 172.21.15.152:0/149630908 learned_addr learned my addr 172.21.15.152:0/149630908 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:50.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.694+0000 7f5241dd7700 1 -- 172.21.15.152:0/149630908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f522c005040 con 0x7f523c0f62a0 2023-12-06T03:54:50.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.694+0000 7f5241dd7700 1 --2- 172.21.15.152:0/149630908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f523c0f62a0 0x7f523c1064b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f522c009830 tx=0x7f522c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:50.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.694+0000 7f52327fc700 1 -- 172.21.15.152:0/149630908 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f522c009320 con 0x7f523c0f62a0 2023-12-06T03:54:50.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.694+0000 7f52327fc700 1 -- 172.21.15.152:0/149630908 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f522c0136c0 con 0x7f523c0f62a0 2023-12-06T03:54:50.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.694+0000 7f5242dd9700 1 -- 172.21.15.152:0/149630908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f523c106ba0 con 0x7f523c0f62a0 2023-12-06T03:54:50.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.694+0000 7f52327fc700 1 -- 172.21.15.152:0/149630908 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f522c01ee90 con 0x7f523c0f62a0 2023-12-06T03:54:50.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.694+0000 7f5242dd9700 1 -- 172.21.15.152:0/149630908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523c107030 con 0x7f523c0f62a0 2023-12-06T03:54:50.697 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.695+0000 7f52327fc700 1 -- 172.21.15.152:0/149630908 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f522c014440 con 0x7f523c0f62a0 2023-12-06T03:54:50.697 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.695+0000 7f52327fc700 1 --2- 172.21.15.152:0/149630908 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f522802ff10 0x7f52280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:50.697 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.695+0000 7f52327fc700 1 -- 172.21.15.152:0/149630908 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f522c041310 con 0x7f523c0f62a0 2023-12-06T03:54:50.697 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.695+0000 7f5242dd9700 1 -- 172.21.15.152:0/149630908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f523c045ad0 con 0x7f523c0f62a0 2023-12-06T03:54:50.697 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.695+0000 7f52415d6700 1 --2- 172.21.15.152:0/149630908 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f522802ff10 0x7f52280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:50.698 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.696+0000 7f52415d6700 1 --2- 172.21.15.152:0/149630908 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f522802ff10 0x7f52280323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f52380096a0 tx=0x7f5238006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:50.701 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.700+0000 7f52327fc700 1 -- 172.21.15.152:0/149630908 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f522c01c500 con 0x7f523c0f62a0 2023-12-06T03:54:50.910 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.909+0000 7f5242dd9700 1 -- 172.21.15.152:0/149630908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f523c107ee0 con 0x7f523c0f62a0 2023-12-06T03:54:50.911 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.910+0000 7f52327fc700 1 -- 172.21.15.152:0/149630908 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f522c01c320 con 0x7f523c0f62a0 2023-12-06T03:54:50.911 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:50.911 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:50.912 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.912+0000 7f5242dd9700 1 -- 172.21.15.152:0/149630908 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f522802ff10 msgr2=0x7f52280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:50.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.912+0000 7f5242dd9700 1 --2- 172.21.15.152:0/149630908 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f522802ff10 0x7f52280323d0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f52380096a0 tx=0x7f5238006b40).stop 2023-12-06T03:54:50.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.912+0000 7f5242dd9700 1 -- 172.21.15.152:0/149630908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f523c0f62a0 msgr2=0x7f523c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:50.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.912+0000 7f5242dd9700 1 --2- 172.21.15.152:0/149630908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f523c0f62a0 0x7f523c1064b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f522c009830 tx=0x7f522c00b040).stop 2023-12-06T03:54:50.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.912+0000 7f5242dd9700 1 -- 172.21.15.152:0/149630908 shutdown_connections 2023-12-06T03:54:50.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.912+0000 7f5242dd9700 1 --2- 172.21.15.152:0/149630908 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f522802ff10 0x7f52280323d0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:50.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.912+0000 7f5242dd9700 1 --2- 172.21.15.152:0/149630908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f523c0f62a0 0x7f523c1064b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:50.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.912+0000 7f5242dd9700 1 -- 172.21.15.152:0/149630908 >> 172.21.15.152:0/149630908 conn(0x7f523c0f1aa0 msgr2=0x7f523c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:50.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.912+0000 7f5242dd9700 1 -- 172.21.15.152:0/149630908 shutdown_connections 2023-12-06T03:54:50.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:50.912+0000 7f5242dd9700 1 -- 172.21.15.152:0/149630908 wait complete. 2023-12-06T03:54:50.913 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:52.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:51 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:50.398896+0000 mgr.smithi132.gokxja (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:52.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:51 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:50.910554+0000 mon.smithi132 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.152:0/149630908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:52.243 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:52.244 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:53.352 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.351+0000 7f87c695f700 1 -- 172.21.15.152:0/756709435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c00f5410 msgr2=0x7f87c00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:53.352 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.351+0000 7f87c695f700 1 --2- 172.21.15.152:0/756709435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c00f5410 0x7f87c00f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f87b4009fb0 tx=0x7f87b4009b30).stop 2023-12-06T03:54:53.352 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.351+0000 7f87c695f700 1 -- 172.21.15.152:0/756709435 shutdown_connections 2023-12-06T03:54:53.353 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.351+0000 7f87c695f700 1 --2- 172.21.15.152:0/756709435 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c00f5410 0x7f87c00f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:53.353 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.351+0000 7f87c695f700 1 -- 172.21.15.152:0/756709435 >> 172.21.15.152:0/756709435 conn(0x7f87c00f0b70 msgr2=0x7f87c00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:53.353 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.351+0000 7f87c695f700 1 -- 172.21.15.152:0/756709435 shutdown_connections 2023-12-06T03:54:53.353 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.351+0000 7f87c695f700 1 -- 172.21.15.152:0/756709435 wait complete. 2023-12-06T03:54:53.353 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.351+0000 7f87c695f700 1 Processor -- start 2023-12-06T03:54:53.353 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.352+0000 7f87c695f700 1 -- start start 2023-12-06T03:54:53.353 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.352+0000 7f87c695f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c00f5410 0x7f87c0109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:53.353 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.352+0000 7f87c695f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87c0060220 con 0x7f87c00f5410 2023-12-06T03:54:53.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.352+0000 7f87bffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c00f5410 0x7f87c0109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:53.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.352+0000 7f87bffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c00f5410 0x7f87c0109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:46240/0 (socket says 172.21.15.152:46240) 2023-12-06T03:54:53.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.352+0000 7f87bffff700 1 -- 172.21.15.152:0/3757950603 learned_addr learned my addr 172.21.15.152:0/3757950603 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:53.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.352+0000 7f87bffff700 1 -- 172.21.15.152:0/3757950603 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87b4005040 con 0x7f87c00f5410 2023-12-06T03:54:53.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.352+0000 7f87bffff700 1 --2- 172.21.15.152:0/3757950603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c00f5410 0x7f87c0109450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f87b4013510 tx=0x7f87b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:53.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.352+0000 7f87bcff9700 1 -- 172.21.15.152:0/3757950603 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87b4009320 con 0x7f87c00f5410 2023-12-06T03:54:53.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.352+0000 7f87bcff9700 1 -- 172.21.15.152:0/3757950603 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87b4013f00 con 0x7f87c00f5410 2023-12-06T03:54:53.356 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.352+0000 7f87c695f700 1 -- 172.21.15.152:0/3757950603 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87c0060420 con 0x7f87c00f5410 2023-12-06T03:54:53.356 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.353+0000 7f87bcff9700 1 -- 172.21.15.152:0/3757950603 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87b4014d90 con 0x7f87c00f5410 2023-12-06T03:54:53.356 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.353+0000 7f87c695f700 1 -- 172.21.15.152:0/3757950603 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87c0109b00 con 0x7f87c00f5410 2023-12-06T03:54:53.356 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.353+0000 7f87bcff9700 1 -- 172.21.15.152:0/3757950603 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87b40095f0 con 0x7f87c00f5410 2023-12-06T03:54:53.356 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.353+0000 7f87c695f700 1 -- 172.21.15.152:0/3757950603 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87c0059100 con 0x7f87c00f5410 2023-12-06T03:54:53.356 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.354+0000 7f87bcff9700 1 --2- 172.21.15.152:0/3757950603 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f87a802ffb0 0x7f87a8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:53.356 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.354+0000 7f87bcff9700 1 -- 172.21.15.152:0/3757950603 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87b4041050 con 0x7f87c00f5410 2023-12-06T03:54:53.356 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.354+0000 7f87bf7fe700 1 --2- 172.21.15.152:0/3757950603 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f87a802ffb0 0x7f87a8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:53.356 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.354+0000 7f87bf7fe700 1 --2- 172.21.15.152:0/3757950603 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f87a802ffb0 0x7f87a8032470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f87b00096a0 tx=0x7f87b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:53.359 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.358+0000 7f87bcff9700 1 -- 172.21.15.152:0/3757950603 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87b401d7f0 con 0x7f87c00f5410 2023-12-06T03:54:53.562 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.561+0000 7f87c695f700 1 -- 172.21.15.152:0/3757950603 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87c0045ad0 con 0x7f87c00f5410 2023-12-06T03:54:53.562 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.562+0000 7f87bcff9700 1 -- 172.21.15.152:0/3757950603 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f87b4010230 con 0x7f87c00f5410 2023-12-06T03:54:53.563 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:53.563 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:53.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.564+0000 7f87c695f700 1 -- 172.21.15.152:0/3757950603 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f87a802ffb0 msgr2=0x7f87a8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:53.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.564+0000 7f87c695f700 1 --2- 172.21.15.152:0/3757950603 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f87a802ffb0 0x7f87a8032470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f87b00096a0 tx=0x7f87b0006b40).stop 2023-12-06T03:54:53.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.564+0000 7f87c695f700 1 -- 172.21.15.152:0/3757950603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c00f5410 msgr2=0x7f87c0109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:53.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.564+0000 7f87c695f700 1 --2- 172.21.15.152:0/3757950603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c00f5410 0x7f87c0109450 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f87b4013510 tx=0x7f87b400b040).stop 2023-12-06T03:54:53.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.564+0000 7f87c695f700 1 -- 172.21.15.152:0/3757950603 shutdown_connections 2023-12-06T03:54:53.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.564+0000 7f87c695f700 1 --2- 172.21.15.152:0/3757950603 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f87a802ffb0 0x7f87a8032470 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:53.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.564+0000 7f87c695f700 1 --2- 172.21.15.152:0/3757950603 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f87c00f5410 0x7f87c0109450 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:53.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.564+0000 7f87c695f700 1 -- 172.21.15.152:0/3757950603 >> 172.21.15.152:0/3757950603 conn(0x7f87c00f0b70 msgr2=0x7f87c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:53.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.564+0000 7f87c695f700 1 -- 172.21.15.152:0/3757950603 shutdown_connections 2023-12-06T03:54:53.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:53.564+0000 7f87c695f700 1 -- 172.21.15.152:0/3757950603 wait complete. 2023-12-06T03:54:53.566 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:54.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:53 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:52.399280+0000 mgr.smithi132.gokxja (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:54.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:53 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:53.562367+0000 mon.smithi132 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.152:0/3757950603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:54.933 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:54.934 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:55.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:54.517327+0000 mon.smithi132 (mon.0) 152 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/mirror_snapshot_schedule"}]: dispatch 2023-12-06T03:54:55.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:54.526146+0000 mon.smithi132 (mon.0) 153 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/trash_purge_schedule"}]: dispatch 2023-12-06T03:54:56.016 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.014+0000 7f132881b700 1 -- 172.21.15.152:0/2553352395 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13240f6510 msgr2=0x7f13240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:56.016 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.014+0000 7f132881b700 1 --2- 172.21.15.152:0/2553352395 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13240f6510 0x7f13240f6930 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f1314009960 tx=0x7f1314004d10).stop 2023-12-06T03:54:56.016 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.014+0000 7f132881b700 1 -- 172.21.15.152:0/2553352395 shutdown_connections 2023-12-06T03:54:56.016 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.014+0000 7f132881b700 1 --2- 172.21.15.152:0/2553352395 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13240f6510 0x7f13240f6930 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:56.017 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.014+0000 7f132881b700 1 -- 172.21.15.152:0/2553352395 >> 172.21.15.152:0/2553352395 conn(0x7f13240f1c70 msgr2=0x7f13240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:56.017 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.014+0000 7f132881b700 1 -- 172.21.15.152:0/2553352395 shutdown_connections 2023-12-06T03:54:56.017 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.014+0000 7f132881b700 1 -- 172.21.15.152:0/2553352395 wait complete. 2023-12-06T03:54:56.017 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.014+0000 7f132881b700 1 Processor -- start 2023-12-06T03:54:56.017 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.014+0000 7f132881b700 1 -- start start 2023-12-06T03:54:56.017 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.015+0000 7f132881b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13240f6510 0x7f1324106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:56.017 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.015+0000 7f132881b700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1324106c30 con 0x7f13240f6510 2023-12-06T03:54:56.017 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.015+0000 7f132259c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13240f6510 0x7f1324106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:56.017 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.015+0000 7f132259c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13240f6510 0x7f1324106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:46254/0 (socket says 172.21.15.152:46254) 2023-12-06T03:54:56.017 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.015+0000 7f132259c700 1 -- 172.21.15.152:0/1696031496 learned_addr learned my addr 172.21.15.152:0/1696031496 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:56.017 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.015+0000 7f132259c700 1 -- 172.21.15.152:0/1696031496 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1314009640 con 0x7f13240f6510 2023-12-06T03:54:56.017 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.015+0000 7f132259c700 1 --2- 172.21.15.152:0/1696031496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13240f6510 0x7f1324106740 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f1314004a10 tx=0x7f1314004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:56.017 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.015+0000 7f131affd700 1 -- 172.21.15.152:0/1696031496 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1314018070 con 0x7f13240f6510 2023-12-06T03:54:56.018 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.015+0000 7f131affd700 1 -- 172.21.15.152:0/1696031496 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1314007e70 con 0x7f13240f6510 2023-12-06T03:54:56.018 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.015+0000 7f132881b700 1 -- 172.21.15.152:0/1696031496 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1324106e30 con 0x7f13240f6510 2023-12-06T03:54:56.018 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.015+0000 7f131affd700 1 -- 172.21.15.152:0/1696031496 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f131401d420 con 0x7f13240f6510 2023-12-06T03:54:56.018 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.015+0000 7f132881b700 1 -- 172.21.15.152:0/1696031496 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13241072c0 con 0x7f13240f6510 2023-12-06T03:54:56.019 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.016+0000 7f131affd700 1 -- 172.21.15.152:0/1696031496 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1314013630 con 0x7f13240f6510 2023-12-06T03:54:56.019 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.016+0000 7f132881b700 1 -- 172.21.15.152:0/1696031496 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1324045ad0 con 0x7f13240f6510 2023-12-06T03:54:56.019 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.016+0000 7f131affd700 1 --2- 172.21.15.152:0/1696031496 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f131002ff60 0x7f1310032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:56.019 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.016+0000 7f131affd700 1 -- 172.21.15.152:0/1696031496 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1314041680 con 0x7f13240f6510 2023-12-06T03:54:56.019 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.017+0000 7f1321d9b700 1 --2- 172.21.15.152:0/1696031496 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f131002ff60 0x7f1310032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:56.019 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.017+0000 7f1321d9b700 1 --2- 172.21.15.152:0/1696031496 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f131002ff60 0x7f1310032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f130c0096a0 tx=0x7f130c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:56.022 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.021+0000 7f131affd700 1 -- 172.21.15.152:0/1696031496 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f131401bb60 con 0x7f13240f6510 2023-12-06T03:54:56.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:54.399622+0000 mgr.smithi132.gokxja (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:56.227 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.227+0000 7f132881b700 1 -- 172.21.15.152:0/1696031496 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13240fbf60 con 0x7f13240f6510 2023-12-06T03:54:56.228 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.227+0000 7f131affd700 1 -- 172.21.15.152:0/1696031496 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f131400b870 con 0x7f13240f6510 2023-12-06T03:54:56.228 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:56.228 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:56.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.229+0000 7f132881b700 1 -- 172.21.15.152:0/1696031496 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f131002ff60 msgr2=0x7f1310032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:56.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.229+0000 7f132881b700 1 --2- 172.21.15.152:0/1696031496 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f131002ff60 0x7f1310032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f130c0096a0 tx=0x7f130c006b40).stop 2023-12-06T03:54:56.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.229+0000 7f132881b700 1 -- 172.21.15.152:0/1696031496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13240f6510 msgr2=0x7f1324106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:56.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.229+0000 7f132881b700 1 --2- 172.21.15.152:0/1696031496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13240f6510 0x7f1324106740 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f1314004a10 tx=0x7f1314004d10).stop 2023-12-06T03:54:56.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.229+0000 7f132881b700 1 -- 172.21.15.152:0/1696031496 shutdown_connections 2023-12-06T03:54:56.231 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.229+0000 7f132881b700 1 --2- 172.21.15.152:0/1696031496 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f131002ff60 0x7f1310032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:56.231 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.229+0000 7f132881b700 1 --2- 172.21.15.152:0/1696031496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13240f6510 0x7f1324106740 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:56.231 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.229+0000 7f132881b700 1 -- 172.21.15.152:0/1696031496 >> 172.21.15.152:0/1696031496 conn(0x7f13240f1c70 msgr2=0x7f13240f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:56.231 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.229+0000 7f132881b700 1 -- 172.21.15.152:0/1696031496 shutdown_connections 2023-12-06T03:54:56.231 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:56.229+0000 7f132881b700 1 -- 172.21.15.152:0/1696031496 wait complete. 2023-12-06T03:54:56.231 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:54:57.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:56 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:56.227843+0000 mon.smithi132 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.152:0/1696031496' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:54:57.618 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:54:57.618 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:54:58.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:56.400046+0000 mgr.smithi132.gokxja (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:54:58.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.663+0000 7f6c0afd9700 1 -- 172.21.15.152:0/4205250947 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c040f5410 msgr2=0x7f6c040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:58.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.663+0000 7f6c0afd9700 1 --2- 172.21.15.152:0/4205250947 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c040f5410 0x7f6c040f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f6bf8009fb0 tx=0x7f6bf8009b30).stop 2023-12-06T03:54:58.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.663+0000 7f6c0afd9700 1 -- 172.21.15.152:0/4205250947 shutdown_connections 2023-12-06T03:54:58.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.663+0000 7f6c0afd9700 1 --2- 172.21.15.152:0/4205250947 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c040f5410 0x7f6c040f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:58.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.663+0000 7f6c0afd9700 1 -- 172.21.15.152:0/4205250947 >> 172.21.15.152:0/4205250947 conn(0x7f6c040f0b70 msgr2=0x7f6c040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:58.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.663+0000 7f6c0afd9700 1 -- 172.21.15.152:0/4205250947 shutdown_connections 2023-12-06T03:54:58.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.663+0000 7f6c0afd9700 1 -- 172.21.15.152:0/4205250947 wait complete. 2023-12-06T03:54:58.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.664+0000 7f6c0afd9700 1 Processor -- start 2023-12-06T03:54:58.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.664+0000 7f6c0afd9700 1 -- start start 2023-12-06T03:54:58.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.664+0000 7f6c0afd9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c040f5410 0x7f6c04105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:58.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.664+0000 7f6c0afd9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c04105b20 con 0x7f6c040f5410 2023-12-06T03:54:58.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.664+0000 7f6c08d75700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c040f5410 0x7f6c04105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:58.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.664+0000 7f6c08d75700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c040f5410 0x7f6c04105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:46268/0 (socket says 172.21.15.152:46268) 2023-12-06T03:54:58.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.664+0000 7f6c08d75700 1 -- 172.21.15.152:0/1994427740 learned_addr learned my addr 172.21.15.152:0/1994427740 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:54:58.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.665+0000 7f6c08d75700 1 -- 172.21.15.152:0/1994427740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf8005040 con 0x7f6c040f5410 2023-12-06T03:54:58.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.665+0000 7f6c08d75700 1 --2- 172.21.15.152:0/1994427740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c040f5410 0x7f6c04105630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6bf8013510 tx=0x7f6bf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:58.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.665+0000 7f6c017fa700 1 -- 172.21.15.152:0/1994427740 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf8009320 con 0x7f6c040f5410 2023-12-06T03:54:58.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.665+0000 7f6c0afd9700 1 -- 172.21.15.152:0/1994427740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c04105d20 con 0x7f6c040f5410 2023-12-06T03:54:58.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.665+0000 7f6c017fa700 1 -- 172.21.15.152:0/1994427740 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bf8013f00 con 0x7f6c040f5410 2023-12-06T03:54:58.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.665+0000 7f6c017fa700 1 -- 172.21.15.152:0/1994427740 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf8014d90 con 0x7f6c040f5410 2023-12-06T03:54:58.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.665+0000 7f6c0afd9700 1 -- 172.21.15.152:0/1994427740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c041061b0 con 0x7f6c040f5410 2023-12-06T03:54:58.668 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.666+0000 7f6c017fa700 1 -- 172.21.15.152:0/1994427740 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6bf800f410 con 0x7f6c040f5410 2023-12-06T03:54:58.668 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.666+0000 7f6c017fa700 1 --2- 172.21.15.152:0/1994427740 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f6bf402ff60 0x7f6bf4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:54:58.668 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.666+0000 7f6c017fa700 1 -- 172.21.15.152:0/1994427740 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6bf8042e10 con 0x7f6c040f5410 2023-12-06T03:54:58.668 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.666+0000 7f6c0afd9700 1 -- 172.21.15.152:0/1994427740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c04045ad0 con 0x7f6c040f5410 2023-12-06T03:54:58.668 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.666+0000 7f6c03fff700 1 --2- 172.21.15.152:0/1994427740 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f6bf402ff60 0x7f6bf4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:54:58.668 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.667+0000 7f6c03fff700 1 --2- 172.21.15.152:0/1994427740 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f6bf402ff60 0x7f6bf4032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f6bf00096a0 tx=0x7f6bf0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:54:58.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.671+0000 7f6c017fa700 1 -- 172.21.15.152:0/1994427740 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bf80283a0 con 0x7f6c040f5410 2023-12-06T03:54:58.878 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.877+0000 7f6c0afd9700 1 -- 172.21.15.152:0/1994427740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c040faf50 con 0x7f6c040f5410 2023-12-06T03:54:58.879 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.878+0000 7f6c017fa700 1 -- 172.21.15.152:0/1994427740 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6bf8012980 con 0x7f6c040f5410 2023-12-06T03:54:58.879 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:54:58.879 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:54:58.880 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.880+0000 7f6c0afd9700 1 -- 172.21.15.152:0/1994427740 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f6bf402ff60 msgr2=0x7f6bf4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:58.880 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.880+0000 7f6c0afd9700 1 --2- 172.21.15.152:0/1994427740 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f6bf402ff60 0x7f6bf4032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f6bf00096a0 tx=0x7f6bf0006b40).stop 2023-12-06T03:54:58.881 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.880+0000 7f6c0afd9700 1 -- 172.21.15.152:0/1994427740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c040f5410 msgr2=0x7f6c04105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:54:58.881 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.880+0000 7f6c0afd9700 1 --2- 172.21.15.152:0/1994427740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c040f5410 0x7f6c04105630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f6bf8013510 tx=0x7f6bf800b040).stop 2023-12-06T03:54:58.881 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.880+0000 7f6c0afd9700 1 -- 172.21.15.152:0/1994427740 shutdown_connections 2023-12-06T03:54:58.881 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.880+0000 7f6c0afd9700 1 --2- 172.21.15.152:0/1994427740 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f6bf402ff60 0x7f6bf4032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:58.881 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.880+0000 7f6c0afd9700 1 --2- 172.21.15.152:0/1994427740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c040f5410 0x7f6c04105630 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:54:58.881 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.880+0000 7f6c0afd9700 1 -- 172.21.15.152:0/1994427740 >> 172.21.15.152:0/1994427740 conn(0x7f6c040f0b70 msgr2=0x7f6c040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:54:58.881 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.880+0000 7f6c0afd9700 1 -- 172.21.15.152:0/1994427740 shutdown_connections 2023-12-06T03:54:58.881 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:54:58.880+0000 7f6c0afd9700 1 -- 172.21.15.152:0/1994427740 wait complete. 2023-12-06T03:54:58.881 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:00.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:59 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:54:58.400362+0000 mgr.smithi132.gokxja (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:00.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:54:59 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:54:58.878433+0000 mon.smithi132 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.152:0/1994427740' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:00.253 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:00.254 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:01.410 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.408+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1180966874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c4060220 msgr2=0x7f13c40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:01.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.408+0000 7f13cb6f5700 1 --2- 172.21.15.152:0/1180966874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c4060220 0x7f13c40f9150 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f13b4009fb0 tx=0x7f13b4009b30).stop 2023-12-06T03:55:01.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.408+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1180966874 shutdown_connections 2023-12-06T03:55:01.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.408+0000 7f13cb6f5700 1 --2- 172.21.15.152:0/1180966874 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c4060220 0x7f13c40f9150 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:01.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.408+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1180966874 >> 172.21.15.152:0/1180966874 conn(0x7f13c40f0b70 msgr2=0x7f13c40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:01.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.408+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1180966874 shutdown_connections 2023-12-06T03:55:01.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.408+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1180966874 wait complete. 2023-12-06T03:55:01.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.409+0000 7f13cb6f5700 1 Processor -- start 2023-12-06T03:55:01.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.409+0000 7f13cb6f5700 1 -- start start 2023-12-06T03:55:01.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.409+0000 7f13cb6f5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c4060220 0x7f13c40f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:01.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.409+0000 7f13cb6f5700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13c40f5050 con 0x7f13c4060220 2023-12-06T03:55:01.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.409+0000 7f13c9491700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c4060220 0x7f13c40f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:01.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.409+0000 7f13c9491700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c4060220 0x7f13c40f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:36522/0 (socket says 172.21.15.152:36522) 2023-12-06T03:55:01.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.409+0000 7f13c9491700 1 -- 172.21.15.152:0/1614814726 learned_addr learned my addr 172.21.15.152:0/1614814726 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:01.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.410+0000 7f13c9491700 1 -- 172.21.15.152:0/1614814726 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13b4005040 con 0x7f13c4060220 2023-12-06T03:55:01.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.410+0000 7f13c9491700 1 --2- 172.21.15.152:0/1614814726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c4060220 0x7f13c40f6bd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f13b4013510 tx=0x7f13b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:01.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.410+0000 7f13b9ffb700 1 -- 172.21.15.152:0/1614814726 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13b4009320 con 0x7f13c4060220 2023-12-06T03:55:01.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.410+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1614814726 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13c40f51f0 con 0x7f13c4060220 2023-12-06T03:55:01.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.410+0000 7f13b9ffb700 1 -- 172.21.15.152:0/1614814726 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13b4013f00 con 0x7f13c4060220 2023-12-06T03:55:01.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.410+0000 7f13b9ffb700 1 -- 172.21.15.152:0/1614814726 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13b4014dc0 con 0x7f13c4060220 2023-12-06T03:55:01.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.410+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1614814726 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13c40f5680 con 0x7f13c4060220 2023-12-06T03:55:01.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.411+0000 7f13b9ffb700 1 -- 172.21.15.152:0/1614814726 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13b40095f0 con 0x7f13c4060220 2023-12-06T03:55:01.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.411+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1614814726 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13c4059100 con 0x7f13c4060220 2023-12-06T03:55:01.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.411+0000 7f13b9ffb700 1 --2- 172.21.15.152:0/1614814726 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f13b002ff60 0x7f13b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:01.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.411+0000 7f13b9ffb700 1 -- 172.21.15.152:0/1614814726 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f13b4041100 con 0x7f13c4060220 2023-12-06T03:55:01.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.411+0000 7f13c8c90700 1 --2- 172.21.15.152:0/1614814726 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f13b002ff60 0x7f13b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:01.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.412+0000 7f13c8c90700 1 --2- 172.21.15.152:0/1614814726 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f13b002ff60 0x7f13b0032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f13c00096a0 tx=0x7f13c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:01.416 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.416+0000 7f13b9ffb700 1 -- 172.21.15.152:0/1614814726 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13b4014420 con 0x7f13c4060220 2023-12-06T03:55:01.624 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.624+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1614814726 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13c40fbf50 con 0x7f13c4060220 2023-12-06T03:55:01.625 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.625+0000 7f13b9ffb700 1 -- 172.21.15.152:0/1614814726 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f13b4010230 con 0x7f13c4060220 2023-12-06T03:55:01.625 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:01.625 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:01.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.626+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1614814726 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f13b002ff60 msgr2=0x7f13b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:01.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.626+0000 7f13cb6f5700 1 --2- 172.21.15.152:0/1614814726 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f13b002ff60 0x7f13b0032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f13c00096a0 tx=0x7f13c0006b40).stop 2023-12-06T03:55:01.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.626+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1614814726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c4060220 msgr2=0x7f13c40f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:01.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.626+0000 7f13cb6f5700 1 --2- 172.21.15.152:0/1614814726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c4060220 0x7f13c40f6bd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f13b4013510 tx=0x7f13b400b040).stop 2023-12-06T03:55:01.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.626+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1614814726 shutdown_connections 2023-12-06T03:55:01.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.626+0000 7f13cb6f5700 1 --2- 172.21.15.152:0/1614814726 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f13b002ff60 0x7f13b0032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:01.628 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.626+0000 7f13cb6f5700 1 --2- 172.21.15.152:0/1614814726 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f13c4060220 0x7f13c40f6bd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:01.628 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.626+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1614814726 >> 172.21.15.152:0/1614814726 conn(0x7f13c40f0b70 msgr2=0x7f13c40f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:01.628 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.626+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1614814726 shutdown_connections 2023-12-06T03:55:01.628 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:01.626+0000 7f13cb6f5700 1 -- 172.21.15.152:0/1614814726 wait complete. 2023-12-06T03:55:01.628 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:02.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:01 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:00.400780+0000 mgr.smithi132.gokxja (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:02.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:01 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:01.624923+0000 mon.smithi132 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.152:0/1614814726' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:03.004 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:03.005 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:04.096 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.095+0000 7f93899a6700 1 -- 172.21.15.152:0/3380949804 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93840f6510 msgr2=0x7f93840f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:04.096 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.095+0000 7f93899a6700 1 --2- 172.21.15.152:0/3380949804 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93840f6510 0x7f93840f6930 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f9378009fb0 tx=0x7f9378009b30).stop 2023-12-06T03:55:04.096 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.095+0000 7f93899a6700 1 -- 172.21.15.152:0/3380949804 shutdown_connections 2023-12-06T03:55:04.096 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.095+0000 7f93899a6700 1 --2- 172.21.15.152:0/3380949804 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93840f6510 0x7f93840f6930 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:04.096 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.095+0000 7f93899a6700 1 -- 172.21.15.152:0/3380949804 >> 172.21.15.152:0/3380949804 conn(0x7f93840f1c70 msgr2=0x7f93840f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:04.097 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.095+0000 7f93899a6700 1 -- 172.21.15.152:0/3380949804 shutdown_connections 2023-12-06T03:55:04.097 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.095+0000 7f93899a6700 1 -- 172.21.15.152:0/3380949804 wait complete. 2023-12-06T03:55:04.098 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.096+0000 7f93899a6700 1 Processor -- start 2023-12-06T03:55:04.098 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.096+0000 7f93899a6700 1 -- start start 2023-12-06T03:55:04.098 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.096+0000 7f93899a6700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93840f6510 0x7f9384106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:04.098 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.096+0000 7f93899a6700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9384106c20 con 0x7f93840f6510 2023-12-06T03:55:04.098 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.096+0000 7f9382ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93840f6510 0x7f9384106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:04.098 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.097+0000 7f9382ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93840f6510 0x7f9384106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:36530/0 (socket says 172.21.15.152:36530) 2023-12-06T03:55:04.098 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.097+0000 7f9382ffd700 1 -- 172.21.15.152:0/3527402114 learned_addr learned my addr 172.21.15.152:0/3527402114 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:04.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.097+0000 7f9382ffd700 1 -- 172.21.15.152:0/3527402114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9378005040 con 0x7f93840f6510 2023-12-06T03:55:04.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.097+0000 7f9382ffd700 1 --2- 172.21.15.152:0/3527402114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93840f6510 0x7f9384106730 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f9378013510 tx=0x7f937800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:04.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.097+0000 7f936b7fe700 1 -- 172.21.15.152:0/3527402114 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9378009320 con 0x7f93840f6510 2023-12-06T03:55:04.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.097+0000 7f93899a6700 1 -- 172.21.15.152:0/3527402114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9384106e20 con 0x7f93840f6510 2023-12-06T03:55:04.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.097+0000 7f936b7fe700 1 -- 172.21.15.152:0/3527402114 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9378013f00 con 0x7f93840f6510 2023-12-06T03:55:04.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.097+0000 7f936b7fe700 1 -- 172.21.15.152:0/3527402114 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9378014d90 con 0x7f93840f6510 2023-12-06T03:55:04.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.097+0000 7f93899a6700 1 -- 172.21.15.152:0/3527402114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93841072b0 con 0x7f93840f6510 2023-12-06T03:55:04.100 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.098+0000 7f936b7fe700 1 -- 172.21.15.152:0/3527402114 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9378014420 con 0x7f93840f6510 2023-12-06T03:55:04.100 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.098+0000 7f93899a6700 1 -- 172.21.15.152:0/3527402114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9384045ad0 con 0x7f93840f6510 2023-12-06T03:55:04.100 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.098+0000 7f936b7fe700 1 --2- 172.21.15.152:0/3527402114 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f936c02ff60 0x7f936c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:04.100 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.098+0000 7f936b7fe700 1 -- 172.21.15.152:0/3527402114 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9378041080 con 0x7f93840f6510 2023-12-06T03:55:04.100 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.098+0000 7f93827fc700 1 --2- 172.21.15.152:0/3527402114 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f936c02ff60 0x7f936c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:04.100 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.099+0000 7f93827fc700 1 --2- 172.21.15.152:0/3527402114 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f936c02ff60 0x7f936c032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f93740096a0 tx=0x7f9374006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:04.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.103+0000 7f936b7fe700 1 -- 172.21.15.152:0/3527402114 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93780095f0 con 0x7f93840f6510 2023-12-06T03:55:04.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:02.401091+0000 mgr.smithi132.gokxja (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:04.310 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.310+0000 7f93899a6700 1 -- 172.21.15.152:0/3527402114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93840fbf60 con 0x7f93840f6510 2023-12-06T03:55:04.311 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.310+0000 7f936b7fe700 1 -- 172.21.15.152:0/3527402114 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f937801db70 con 0x7f93840f6510 2023-12-06T03:55:04.311 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:04.311 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:04.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.312+0000 7f93899a6700 1 -- 172.21.15.152:0/3527402114 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f936c02ff60 msgr2=0x7f936c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:04.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.312+0000 7f93899a6700 1 --2- 172.21.15.152:0/3527402114 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f936c02ff60 0x7f936c032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f93740096a0 tx=0x7f9374006b40).stop 2023-12-06T03:55:04.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.312+0000 7f93899a6700 1 -- 172.21.15.152:0/3527402114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93840f6510 msgr2=0x7f9384106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:04.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.312+0000 7f93899a6700 1 --2- 172.21.15.152:0/3527402114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93840f6510 0x7f9384106730 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f9378013510 tx=0x7f937800b040).stop 2023-12-06T03:55:04.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.312+0000 7f93899a6700 1 -- 172.21.15.152:0/3527402114 shutdown_connections 2023-12-06T03:55:04.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.312+0000 7f93899a6700 1 --2- 172.21.15.152:0/3527402114 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f936c02ff60 0x7f936c032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:04.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.312+0000 7f93899a6700 1 --2- 172.21.15.152:0/3527402114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f93840f6510 0x7f9384106730 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:04.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.312+0000 7f93899a6700 1 -- 172.21.15.152:0/3527402114 >> 172.21.15.152:0/3527402114 conn(0x7f93840f1c70 msgr2=0x7f93840f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:04.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.312+0000 7f93899a6700 1 -- 172.21.15.152:0/3527402114 shutdown_connections 2023-12-06T03:55:04.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:04.312+0000 7f93899a6700 1 -- 172.21.15.152:0/3527402114 wait complete. 2023-12-06T03:55:04.314 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:05.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:04 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:04.310824+0000 mon.smithi132 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.152:0/3527402114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:05.687 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:05.688 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:06.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:04.401372+0000 mgr.smithi132.gokxja (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:06.910 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.907+0000 7fb507951700 1 -- 172.21.15.152:0/508884817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5000f5410 msgr2=0x7fb5000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:06.911 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.907+0000 7fb507951700 1 --2- 172.21.15.152:0/508884817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5000f5410 0x7fb5000f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7fb4f4009fb0 tx=0x7fb4f4009b30).stop 2023-12-06T03:55:06.911 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.907+0000 7fb507951700 1 -- 172.21.15.152:0/508884817 shutdown_connections 2023-12-06T03:55:06.911 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.907+0000 7fb507951700 1 --2- 172.21.15.152:0/508884817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5000f5410 0x7fb5000f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:06.911 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.907+0000 7fb507951700 1 -- 172.21.15.152:0/508884817 >> 172.21.15.152:0/508884817 conn(0x7fb5000f0b70 msgr2=0x7fb5000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:06.911 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.907+0000 7fb507951700 1 -- 172.21.15.152:0/508884817 shutdown_connections 2023-12-06T03:55:06.911 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.907+0000 7fb507951700 1 -- 172.21.15.152:0/508884817 wait complete. 2023-12-06T03:55:06.911 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.907+0000 7fb507951700 1 Processor -- start 2023-12-06T03:55:06.911 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.908+0000 7fb507951700 1 -- start start 2023-12-06T03:55:06.911 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.908+0000 7fb507951700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5000f5410 0x7fb5000fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:06.911 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.908+0000 7fb507951700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5000fb820 con 0x7fb5000f5410 2023-12-06T03:55:06.911 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.908+0000 7fb5056ed700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5000f5410 0x7fb5000fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:06.912 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.908+0000 7fb5056ed700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5000f5410 0x7fb5000fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:36552/0 (socket says 172.21.15.152:36552) 2023-12-06T03:55:06.912 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.908+0000 7fb5056ed700 1 -- 172.21.15.152:0/1284699623 learned_addr learned my addr 172.21.15.152:0/1284699623 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:06.912 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.908+0000 7fb5056ed700 1 -- 172.21.15.152:0/1284699623 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4f4005040 con 0x7fb5000f5410 2023-12-06T03:55:06.912 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.908+0000 7fb5056ed700 1 --2- 172.21.15.152:0/1284699623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5000f5410 0x7fb5000fd3a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fb4f4013510 tx=0x7fb4f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:06.912 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.908+0000 7fb4f1ffb700 1 -- 172.21.15.152:0/1284699623 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4f4009320 con 0x7fb5000f5410 2023-12-06T03:55:06.912 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.908+0000 7fb4f1ffb700 1 -- 172.21.15.152:0/1284699623 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb4f4013f00 con 0x7fb5000f5410 2023-12-06T03:55:06.912 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.908+0000 7fb507951700 1 -- 172.21.15.152:0/1284699623 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5000fb9c0 con 0x7fb5000f5410 2023-12-06T03:55:06.912 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.908+0000 7fb4f1ffb700 1 -- 172.21.15.152:0/1284699623 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb4f4014d90 con 0x7fb5000f5410 2023-12-06T03:55:06.912 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.909+0000 7fb507951700 1 -- 172.21.15.152:0/1284699623 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5000fbe50 con 0x7fb5000f5410 2023-12-06T03:55:06.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.909+0000 7fb4f1ffb700 1 -- 172.21.15.152:0/1284699623 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4f40095f0 con 0x7fb5000f5410 2023-12-06T03:55:06.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.909+0000 7fb507951700 1 -- 172.21.15.152:0/1284699623 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb500059100 con 0x7fb5000f5410 2023-12-06T03:55:06.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.910+0000 7fb4f1ffb700 1 --2- 172.21.15.152:0/1284699623 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb4ec02ff60 0x7fb4ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:06.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.910+0000 7fb4f1ffb700 1 -- 172.21.15.152:0/1284699623 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4f4041000 con 0x7fb5000f5410 2023-12-06T03:55:06.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.910+0000 7fb504eec700 1 --2- 172.21.15.152:0/1284699623 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb4ec02ff60 0x7fb4ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:06.913 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.910+0000 7fb504eec700 1 --2- 172.21.15.152:0/1284699623 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb4ec02ff60 0x7fb4ec032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fb4fc0096a0 tx=0x7fb4fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:06.916 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:06.914+0000 7fb4f1ffb700 1 -- 172.21.15.152:0/1284699623 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4f4014420 con 0x7fb5000f5410 2023-12-06T03:55:07.121 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:07.120+0000 7fb507951700 1 -- 172.21.15.152:0/1284699623 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb500045ad0 con 0x7fb5000f5410 2023-12-06T03:55:07.122 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:07.120+0000 7fb4f1ffb700 1 -- 172.21.15.152:0/1284699623 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb4f4010230 con 0x7fb5000f5410 2023-12-06T03:55:07.122 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:07.122 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:07.124 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:07.122+0000 7fb507951700 1 -- 172.21.15.152:0/1284699623 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb4ec02ff60 msgr2=0x7fb4ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:07.124 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:07.122+0000 7fb507951700 1 --2- 172.21.15.152:0/1284699623 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb4ec02ff60 0x7fb4ec032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7fb4fc0096a0 tx=0x7fb4fc006b40).stop 2023-12-06T03:55:07.124 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:07.122+0000 7fb507951700 1 -- 172.21.15.152:0/1284699623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5000f5410 msgr2=0x7fb5000fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:07.124 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:07.122+0000 7fb507951700 1 --2- 172.21.15.152:0/1284699623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5000f5410 0x7fb5000fd3a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fb4f4013510 tx=0x7fb4f400b040).stop 2023-12-06T03:55:07.124 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:07.122+0000 7fb507951700 1 -- 172.21.15.152:0/1284699623 shutdown_connections 2023-12-06T03:55:07.124 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:07.122+0000 7fb507951700 1 --2- 172.21.15.152:0/1284699623 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb4ec02ff60 0x7fb4ec032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:07.124 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:07.122+0000 7fb507951700 1 --2- 172.21.15.152:0/1284699623 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb5000f5410 0x7fb5000fd3a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:07.124 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:07.122+0000 7fb507951700 1 -- 172.21.15.152:0/1284699623 >> 172.21.15.152:0/1284699623 conn(0x7fb5000f0b70 msgr2=0x7fb5000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:07.124 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:07.122+0000 7fb507951700 1 -- 172.21.15.152:0/1284699623 shutdown_connections 2023-12-06T03:55:07.124 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:07.122+0000 7fb507951700 1 -- 172.21.15.152:0/1284699623 wait complete. 2023-12-06T03:55:07.125 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:08.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:07 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:06.401772+0000 mgr.smithi132.gokxja (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:08.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:07 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:07.121821+0000 mon.smithi132 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.152:0/1284699623' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:08.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:08.497 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:09.572 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.569+0000 7f70c303e700 1 -- 172.21.15.152:0/557936358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70bc0f51a0 msgr2=0x7f70bc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:09.572 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.569+0000 7f70c303e700 1 --2- 172.21.15.152:0/557936358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70bc0f51a0 0x7f70bc0f55c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f70b8009fb0 tx=0x7f70b8009b30).stop 2023-12-06T03:55:09.572 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.569+0000 7f70c303e700 1 -- 172.21.15.152:0/557936358 shutdown_connections 2023-12-06T03:55:09.572 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.569+0000 7f70c303e700 1 --2- 172.21.15.152:0/557936358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70bc0f51a0 0x7f70bc0f55c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:09.572 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.569+0000 7f70c303e700 1 -- 172.21.15.152:0/557936358 >> 172.21.15.152:0/557936358 conn(0x7f70bc0f09a0 msgr2=0x7f70bc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:09.572 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.569+0000 7f70c303e700 1 -- 172.21.15.152:0/557936358 shutdown_connections 2023-12-06T03:55:09.572 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.569+0000 7f70c303e700 1 -- 172.21.15.152:0/557936358 wait complete. 2023-12-06T03:55:09.572 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.569+0000 7f70c303e700 1 Processor -- start 2023-12-06T03:55:09.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.569+0000 7f70c303e700 1 -- start start 2023-12-06T03:55:09.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.570+0000 7f70c303e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70bc0f51a0 0x7f70bc1064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:09.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.570+0000 7f70c303e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70bc106990 con 0x7f70bc0f51a0 2023-12-06T03:55:09.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.570+0000 7f70c203c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70bc0f51a0 0x7f70bc1064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:09.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.570+0000 7f70c203c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70bc0f51a0 0x7f70bc1064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40618/0 (socket says 172.21.15.152:40618) 2023-12-06T03:55:09.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.570+0000 7f70c203c700 1 -- 172.21.15.152:0/2088296633 learned_addr learned my addr 172.21.15.152:0/2088296633 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:09.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.570+0000 7f70c203c700 1 -- 172.21.15.152:0/2088296633 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b8005040 con 0x7f70bc0f51a0 2023-12-06T03:55:09.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.570+0000 7f70c203c700 1 --2- 172.21.15.152:0/2088296633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70bc0f51a0 0x7f70bc1064a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f70b8013510 tx=0x7f70b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:09.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.570+0000 7f70b2ffd700 1 -- 172.21.15.152:0/2088296633 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b8009320 con 0x7f70bc0f51a0 2023-12-06T03:55:09.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.570+0000 7f70c303e700 1 -- 172.21.15.152:0/2088296633 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70bc106b90 con 0x7f70bc0f51a0 2023-12-06T03:55:09.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.570+0000 7f70b2ffd700 1 -- 172.21.15.152:0/2088296633 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f70b8013f00 con 0x7f70bc0f51a0 2023-12-06T03:55:09.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.570+0000 7f70b2ffd700 1 -- 172.21.15.152:0/2088296633 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b8014dc0 con 0x7f70bc0f51a0 2023-12-06T03:55:09.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.570+0000 7f70c303e700 1 -- 172.21.15.152:0/2088296633 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70bc107020 con 0x7f70bc0f51a0 2023-12-06T03:55:09.574 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.571+0000 7f70b2ffd700 1 -- 172.21.15.152:0/2088296633 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f70b80095f0 con 0x7f70bc0f51a0 2023-12-06T03:55:09.574 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.571+0000 7f70c303e700 1 -- 172.21.15.152:0/2088296633 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70bc045ad0 con 0x7f70bc0f51a0 2023-12-06T03:55:09.574 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.571+0000 7f70b2ffd700 1 --2- 172.21.15.152:0/2088296633 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f70a802ff60 0x7f70a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:09.574 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.571+0000 7f70b2ffd700 1 -- 172.21.15.152:0/2088296633 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f70b8041000 con 0x7f70bc0f51a0 2023-12-06T03:55:09.574 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.572+0000 7f70c183b700 1 --2- 172.21.15.152:0/2088296633 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f70a802ff60 0x7f70a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:09.574 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.572+0000 7f70c183b700 1 --2- 172.21.15.152:0/2088296633 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f70a802ff60 0x7f70a8032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f70ac0096a0 tx=0x7f70ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:09.577 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.576+0000 7f70b2ffd700 1 -- 172.21.15.152:0/2088296633 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f70b8014420 con 0x7f70bc0f51a0 2023-12-06T03:55:09.786 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.784+0000 7f70c303e700 1 -- 172.21.15.152:0/2088296633 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f70bc107ed0 con 0x7f70bc0f51a0 2023-12-06T03:55:09.786 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.784+0000 7f70b2ffd700 1 -- 172.21.15.152:0/2088296633 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f70b8010230 con 0x7f70bc0f51a0 2023-12-06T03:55:09.786 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:09.786 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:09.788 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.786+0000 7f70c303e700 1 -- 172.21.15.152:0/2088296633 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f70a802ff60 msgr2=0x7f70a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:09.788 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.786+0000 7f70c303e700 1 --2- 172.21.15.152:0/2088296633 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f70a802ff60 0x7f70a8032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f70ac0096a0 tx=0x7f70ac006b40).stop 2023-12-06T03:55:09.788 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.786+0000 7f70c303e700 1 -- 172.21.15.152:0/2088296633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70bc0f51a0 msgr2=0x7f70bc1064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:09.788 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.786+0000 7f70c303e700 1 --2- 172.21.15.152:0/2088296633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70bc0f51a0 0x7f70bc1064a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f70b8013510 tx=0x7f70b800b040).stop 2023-12-06T03:55:09.788 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.786+0000 7f70c303e700 1 -- 172.21.15.152:0/2088296633 shutdown_connections 2023-12-06T03:55:09.788 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.786+0000 7f70c303e700 1 --2- 172.21.15.152:0/2088296633 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f70a802ff60 0x7f70a8032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:09.788 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.786+0000 7f70c303e700 1 --2- 172.21.15.152:0/2088296633 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f70bc0f51a0 0x7f70bc1064a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:09.788 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.786+0000 7f70c303e700 1 -- 172.21.15.152:0/2088296633 >> 172.21.15.152:0/2088296633 conn(0x7f70bc0f09a0 msgr2=0x7f70bc05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:09.789 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.786+0000 7f70c303e700 1 -- 172.21.15.152:0/2088296633 shutdown_connections 2023-12-06T03:55:09.789 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:09.786+0000 7f70c303e700 1 -- 172.21.15.152:0/2088296633 wait complete. 2023-12-06T03:55:09.789 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:10.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:09 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:08.402098+0000 mgr.smithi132.gokxja (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:11.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:10 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:09.786025+0000 mon.smithi132 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.152:0/2088296633' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:11.175 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:11.175 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:12.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:11 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:10.402462+0000 mgr.smithi132.gokxja (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:12.229 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.226+0000 7fea0022c700 1 -- 172.21.15.152:0/2893969222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9f80f5410 msgr2=0x7fe9f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:12.229 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.226+0000 7fea0022c700 1 --2- 172.21.15.152:0/2893969222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9f80f5410 0x7fe9f80f5830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fe9e8009fb0 tx=0x7fe9e8009b30).stop 2023-12-06T03:55:12.229 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.226+0000 7fea0022c700 1 -- 172.21.15.152:0/2893969222 shutdown_connections 2023-12-06T03:55:12.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.226+0000 7fea0022c700 1 --2- 172.21.15.152:0/2893969222 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9f80f5410 0x7fe9f80f5830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:12.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.226+0000 7fea0022c700 1 -- 172.21.15.152:0/2893969222 >> 172.21.15.152:0/2893969222 conn(0x7fe9f80f0b70 msgr2=0x7fe9f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:12.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.226+0000 7fea0022c700 1 -- 172.21.15.152:0/2893969222 shutdown_connections 2023-12-06T03:55:12.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.226+0000 7fea0022c700 1 -- 172.21.15.152:0/2893969222 wait complete. 2023-12-06T03:55:12.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.227+0000 7fea0022c700 1 Processor -- start 2023-12-06T03:55:12.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.227+0000 7fea0022c700 1 -- start start 2023-12-06T03:55:12.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.227+0000 7fea0022c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9f80f5410 0x7fe9f81093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:12.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.227+0000 7fea0022c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9f8060280 con 0x7fe9f80f5410 2023-12-06T03:55:12.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.227+0000 7fe9fdfc8700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9f80f5410 0x7fe9f81093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:12.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.227+0000 7fe9fdfc8700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9f80f5410 0x7fe9f81093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40638/0 (socket says 172.21.15.152:40638) 2023-12-06T03:55:12.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.227+0000 7fe9fdfc8700 1 -- 172.21.15.152:0/4081526747 learned_addr learned my addr 172.21.15.152:0/4081526747 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:12.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.227+0000 7fe9fdfc8700 1 -- 172.21.15.152:0/4081526747 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e8005040 con 0x7fe9f80f5410 2023-12-06T03:55:12.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.227+0000 7fe9fdfc8700 1 --2- 172.21.15.152:0/4081526747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9f80f5410 0x7fe9f81093f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fe9e8013510 tx=0x7fe9e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:12.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.227+0000 7fe9ee7fc700 1 -- 172.21.15.152:0/4081526747 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9e8009320 con 0x7fe9f80f5410 2023-12-06T03:55:12.231 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.227+0000 7fe9ee7fc700 1 -- 172.21.15.152:0/4081526747 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9e8013f00 con 0x7fe9f80f5410 2023-12-06T03:55:12.231 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.227+0000 7fea0022c700 1 -- 172.21.15.152:0/4081526747 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9f8060480 con 0x7fe9f80f5410 2023-12-06T03:55:12.231 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.227+0000 7fe9ee7fc700 1 -- 172.21.15.152:0/4081526747 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9e8014d90 con 0x7fe9f80f5410 2023-12-06T03:55:12.231 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.228+0000 7fea0022c700 1 -- 172.21.15.152:0/4081526747 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9f8109b20 con 0x7fe9f80f5410 2023-12-06T03:55:12.232 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.228+0000 7fe9ee7fc700 1 -- 172.21.15.152:0/4081526747 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9e80095f0 con 0x7fe9f80f5410 2023-12-06T03:55:12.232 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.228+0000 7fea0022c700 1 -- 172.21.15.152:0/4081526747 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9f8045ad0 con 0x7fe9f80f5410 2023-12-06T03:55:12.232 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.228+0000 7fe9ee7fc700 1 --2- 172.21.15.152:0/4081526747 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe9e402ff60 0x7fe9e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:12.232 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.228+0000 7fe9ee7fc700 1 -- 172.21.15.152:0/4081526747 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9e80410c0 con 0x7fe9f80f5410 2023-12-06T03:55:12.232 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.229+0000 7fe9fd7c7700 1 --2- 172.21.15.152:0/4081526747 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe9e402ff60 0x7fe9e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:12.232 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.229+0000 7fe9fd7c7700 1 --2- 172.21.15.152:0/4081526747 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe9e402ff60 0x7fe9e4032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fe9f40096a0 tx=0x7fe9f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:12.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.233+0000 7fe9ee7fc700 1 -- 172.21.15.152:0/4081526747 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9e8014420 con 0x7fe9f80f5410 2023-12-06T03:55:12.441 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.439+0000 7fea0022c700 1 -- 172.21.15.152:0/4081526747 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9f810a340 con 0x7fe9f80f5410 2023-12-06T03:55:12.441 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.440+0000 7fe9ee7fc700 1 -- 172.21.15.152:0/4081526747 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe9e801d7f0 con 0x7fe9f80f5410 2023-12-06T03:55:12.441 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:12.442 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:12.443 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.441+0000 7fea0022c700 1 -- 172.21.15.152:0/4081526747 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe9e402ff60 msgr2=0x7fe9e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:12.443 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.441+0000 7fea0022c700 1 --2- 172.21.15.152:0/4081526747 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe9e402ff60 0x7fe9e4032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fe9f40096a0 tx=0x7fe9f4006b40).stop 2023-12-06T03:55:12.443 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.441+0000 7fea0022c700 1 -- 172.21.15.152:0/4081526747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9f80f5410 msgr2=0x7fe9f81093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:12.443 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.441+0000 7fea0022c700 1 --2- 172.21.15.152:0/4081526747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9f80f5410 0x7fe9f81093f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fe9e8013510 tx=0x7fe9e800b040).stop 2023-12-06T03:55:12.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.441+0000 7fea0022c700 1 -- 172.21.15.152:0/4081526747 shutdown_connections 2023-12-06T03:55:12.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.441+0000 7fea0022c700 1 --2- 172.21.15.152:0/4081526747 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe9e402ff60 0x7fe9e4032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:12.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.441+0000 7fea0022c700 1 --2- 172.21.15.152:0/4081526747 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe9f80f5410 0x7fe9f81093f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:12.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.441+0000 7fea0022c700 1 -- 172.21.15.152:0/4081526747 >> 172.21.15.152:0/4081526747 conn(0x7fe9f80f0b70 msgr2=0x7fe9f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:12.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.441+0000 7fea0022c700 1 -- 172.21.15.152:0/4081526747 shutdown_connections 2023-12-06T03:55:12.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:12.441+0000 7fea0022c700 1 -- 172.21.15.152:0/4081526747 wait complete. 2023-12-06T03:55:12.444 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:13.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:12.441129+0000 mon.smithi132 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.152:0/4081526747' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:13.826 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:13.826 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:14.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:13 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:12.402859+0000 mgr.smithi132.gokxja (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:14.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.893+0000 7f9b0846e700 1 -- 172.21.15.152:0/4214185871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b000f5950 msgr2=0x7f9b000f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:14.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.893+0000 7f9b0846e700 1 --2- 172.21.15.152:0/4214185871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b000f5950 0x7f9b000f5d70 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f9af0009fb0 tx=0x7f9af0009b30).stop 2023-12-06T03:55:14.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.893+0000 7f9b0846e700 1 -- 172.21.15.152:0/4214185871 shutdown_connections 2023-12-06T03:55:14.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.893+0000 7f9b0846e700 1 --2- 172.21.15.152:0/4214185871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b000f5950 0x7f9b000f5d70 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:14.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.893+0000 7f9b0846e700 1 -- 172.21.15.152:0/4214185871 >> 172.21.15.152:0/4214185871 conn(0x7f9b000f0260 msgr2=0x7f9b000f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:14.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.893+0000 7f9b0846e700 1 -- 172.21.15.152:0/4214185871 shutdown_connections 2023-12-06T03:55:14.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.893+0000 7f9b0846e700 1 -- 172.21.15.152:0/4214185871 wait complete. 2023-12-06T03:55:14.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.894+0000 7f9b0846e700 1 Processor -- start 2023-12-06T03:55:14.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.894+0000 7f9b0846e700 1 -- start start 2023-12-06T03:55:14.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.894+0000 7f9b0846e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b000f5950 0x7f9b00109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:14.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.894+0000 7f9b0846e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b000f5310 con 0x7f9b000f5950 2023-12-06T03:55:14.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.894+0000 7f9b0620a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b000f5950 0x7f9b00109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:14.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.894+0000 7f9b0620a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b000f5950 0x7f9b00109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40664/0 (socket says 172.21.15.152:40664) 2023-12-06T03:55:14.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.894+0000 7f9b0620a700 1 -- 172.21.15.152:0/1625702205 learned_addr learned my addr 172.21.15.152:0/1625702205 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:14.898 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.894+0000 7f9b0620a700 1 -- 172.21.15.152:0/1625702205 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9af0005040 con 0x7f9b000f5950 2023-12-06T03:55:14.898 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.895+0000 7f9b0620a700 1 --2- 172.21.15.152:0/1625702205 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b000f5950 0x7f9b00109430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f9af0014000 tx=0x7f9af000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:14.898 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.895+0000 7f9af6ffd700 1 -- 172.21.15.152:0/1625702205 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9af00095f0 con 0x7f9b000f5950 2023-12-06T03:55:14.898 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.895+0000 7f9b0846e700 1 -- 172.21.15.152:0/1625702205 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b000f5510 con 0x7f9b000f5950 2023-12-06T03:55:14.898 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.895+0000 7f9af6ffd700 1 -- 172.21.15.152:0/1625702205 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9af0007e70 con 0x7f9b000f5950 2023-12-06T03:55:14.898 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.895+0000 7f9af6ffd700 1 -- 172.21.15.152:0/1625702205 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9af0014ea0 con 0x7f9b000f5950 2023-12-06T03:55:14.898 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.895+0000 7f9b0846e700 1 -- 172.21.15.152:0/1625702205 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b00109ae0 con 0x7f9b000f5950 2023-12-06T03:55:14.899 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.896+0000 7f9b0846e700 1 -- 172.21.15.152:0/1625702205 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b00059100 con 0x7f9b000f5950 2023-12-06T03:55:14.899 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.896+0000 7f9af6ffd700 1 -- 172.21.15.152:0/1625702205 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9af0025070 con 0x7f9b000f5950 2023-12-06T03:55:14.899 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.896+0000 7f9af6ffd700 1 --2- 172.21.15.152:0/1625702205 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9aec02ff60 0x7f9aec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:14.899 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.896+0000 7f9af6ffd700 1 -- 172.21.15.152:0/1625702205 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9af0040940 con 0x7f9b000f5950 2023-12-06T03:55:14.899 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.896+0000 7f9b05a09700 1 --2- 172.21.15.152:0/1625702205 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9aec02ff60 0x7f9aec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:14.899 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.896+0000 7f9b05a09700 1 --2- 172.21.15.152:0/1625702205 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9aec02ff60 0x7f9aec032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f9afc0096a0 tx=0x7f9afc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:14.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:14.901+0000 7f9af6ffd700 1 -- 172.21.15.152:0/1625702205 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9af00079c0 con 0x7f9b000f5950 2023-12-06T03:55:15.109 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:15.107+0000 7f9b0846e700 1 -- 172.21.15.152:0/1625702205 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b000fae40 con 0x7f9b000f5950 2023-12-06T03:55:15.110 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:15.108+0000 7f9af6ffd700 1 -- 172.21.15.152:0/1625702205 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9af000f300 con 0x7f9b000f5950 2023-12-06T03:55:15.110 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:15.110 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:15.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:15.110+0000 7f9b0846e700 1 -- 172.21.15.152:0/1625702205 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9aec02ff60 msgr2=0x7f9aec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:15.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:15.110+0000 7f9b0846e700 1 --2- 172.21.15.152:0/1625702205 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9aec02ff60 0x7f9aec032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f9afc0096a0 tx=0x7f9afc006b40).stop 2023-12-06T03:55:15.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:15.110+0000 7f9b0846e700 1 -- 172.21.15.152:0/1625702205 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b000f5950 msgr2=0x7f9b00109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:15.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:15.110+0000 7f9b0846e700 1 --2- 172.21.15.152:0/1625702205 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b000f5950 0x7f9b00109430 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f9af0014000 tx=0x7f9af000b040).stop 2023-12-06T03:55:15.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:15.110+0000 7f9b0846e700 1 -- 172.21.15.152:0/1625702205 shutdown_connections 2023-12-06T03:55:15.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:15.110+0000 7f9b0846e700 1 --2- 172.21.15.152:0/1625702205 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9aec02ff60 0x7f9aec032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:15.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:15.110+0000 7f9b0846e700 1 --2- 172.21.15.152:0/1625702205 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9b000f5950 0x7f9b00109430 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:15.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:15.110+0000 7f9b0846e700 1 -- 172.21.15.152:0/1625702205 >> 172.21.15.152:0/1625702205 conn(0x7f9b000f0260 msgr2=0x7f9b000f0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:15.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:15.110+0000 7f9b0846e700 1 -- 172.21.15.152:0/1625702205 shutdown_connections 2023-12-06T03:55:15.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:15.110+0000 7f9b0846e700 1 -- 172.21.15.152:0/1625702205 wait complete. 2023-12-06T03:55:15.113 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:16.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:15 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:14.403196+0000 mgr.smithi132.gokxja (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:16.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:15 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:15.109539+0000 mon.smithi132 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.152:0/1625702205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:16.494 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:16.495 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:17.599 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.596+0000 7f2e3412c700 1 -- 172.21.15.152:0/3115420546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e2c0f6510 msgr2=0x7f2e2c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:17.599 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.596+0000 7f2e3412c700 1 --2- 172.21.15.152:0/3115420546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e2c0f6510 0x7f2e2c0f6930 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f2e24009fb0 tx=0x7f2e24009b30).stop 2023-12-06T03:55:17.599 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.596+0000 7f2e3412c700 1 -- 172.21.15.152:0/3115420546 shutdown_connections 2023-12-06T03:55:17.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.596+0000 7f2e3412c700 1 --2- 172.21.15.152:0/3115420546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e2c0f6510 0x7f2e2c0f6930 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:17.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.596+0000 7f2e3412c700 1 -- 172.21.15.152:0/3115420546 >> 172.21.15.152:0/3115420546 conn(0x7f2e2c0f1c70 msgr2=0x7f2e2c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:17.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.596+0000 7f2e3412c700 1 -- 172.21.15.152:0/3115420546 shutdown_connections 2023-12-06T03:55:17.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.596+0000 7f2e3412c700 1 -- 172.21.15.152:0/3115420546 wait complete. 2023-12-06T03:55:17.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.597+0000 7f2e3412c700 1 Processor -- start 2023-12-06T03:55:17.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.597+0000 7f2e3412c700 1 -- start start 2023-12-06T03:55:17.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.597+0000 7f2e3412c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e2c0f6510 0x7f2e2c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:17.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.597+0000 7f2e3412c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e2c106c20 con 0x7f2e2c0f6510 2023-12-06T03:55:17.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.597+0000 7f2e31ec8700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e2c0f6510 0x7f2e2c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:17.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.597+0000 7f2e31ec8700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e2c0f6510 0x7f2e2c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40676/0 (socket says 172.21.15.152:40676) 2023-12-06T03:55:17.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.597+0000 7f2e31ec8700 1 -- 172.21.15.152:0/309493377 learned_addr learned my addr 172.21.15.152:0/309493377 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:17.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.597+0000 7f2e31ec8700 1 -- 172.21.15.152:0/309493377 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e24005040 con 0x7f2e2c0f6510 2023-12-06T03:55:17.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.597+0000 7f2e31ec8700 1 --2- 172.21.15.152:0/309493377 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e2c0f6510 0x7f2e2c106730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f2e24014000 tx=0x7f2e2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:17.601 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.597+0000 7f2e227fc700 1 -- 172.21.15.152:0/309493377 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e240095f0 con 0x7f2e2c0f6510 2023-12-06T03:55:17.601 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.598+0000 7f2e3412c700 1 -- 172.21.15.152:0/309493377 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e2c106e20 con 0x7f2e2c0f6510 2023-12-06T03:55:17.601 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.598+0000 7f2e227fc700 1 -- 172.21.15.152:0/309493377 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e24007e70 con 0x7f2e2c0f6510 2023-12-06T03:55:17.601 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.598+0000 7f2e227fc700 1 -- 172.21.15.152:0/309493377 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e24014ea0 con 0x7f2e2c0f6510 2023-12-06T03:55:17.601 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.598+0000 7f2e3412c700 1 -- 172.21.15.152:0/309493377 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e2c1072b0 con 0x7f2e2c0f6510 2023-12-06T03:55:17.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.598+0000 7f2e227fc700 1 -- 172.21.15.152:0/309493377 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e24025070 con 0x7f2e2c0f6510 2023-12-06T03:55:17.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.598+0000 7f2e3412c700 1 -- 172.21.15.152:0/309493377 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e2c045ad0 con 0x7f2e2c0f6510 2023-12-06T03:55:17.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.598+0000 7f2e227fc700 1 --2- 172.21.15.152:0/309493377 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2e1802ff60 0x7f2e18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:17.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.599+0000 7f2e227fc700 1 -- 172.21.15.152:0/309493377 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e24040a30 con 0x7f2e2c0f6510 2023-12-06T03:55:17.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.599+0000 7f2e316c7700 1 --2- 172.21.15.152:0/309493377 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2e1802ff60 0x7f2e18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:17.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.599+0000 7f2e316c7700 1 --2- 172.21.15.152:0/309493377 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2e1802ff60 0x7f2e18032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f2e1c0096a0 tx=0x7f2e1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:17.605 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.603+0000 7f2e227fc700 1 -- 172.21.15.152:0/309493377 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e240079c0 con 0x7f2e2c0f6510 2023-12-06T03:55:17.815 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.813+0000 7f2e3412c700 1 -- 172.21.15.152:0/309493377 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e2c0fbf60 con 0x7f2e2c0f6510 2023-12-06T03:55:17.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.814+0000 7f2e227fc700 1 -- 172.21.15.152:0/309493377 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2e2400f300 con 0x7f2e2c0f6510 2023-12-06T03:55:17.816 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:17.816 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:17.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.816+0000 7f2e3412c700 1 -- 172.21.15.152:0/309493377 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2e1802ff60 msgr2=0x7f2e18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:17.818 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.816+0000 7f2e3412c700 1 --2- 172.21.15.152:0/309493377 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2e1802ff60 0x7f2e18032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f2e1c0096a0 tx=0x7f2e1c006b40).stop 2023-12-06T03:55:17.818 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.816+0000 7f2e3412c700 1 -- 172.21.15.152:0/309493377 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e2c0f6510 msgr2=0x7f2e2c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:17.818 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.816+0000 7f2e3412c700 1 --2- 172.21.15.152:0/309493377 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e2c0f6510 0x7f2e2c106730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f2e24014000 tx=0x7f2e2400b040).stop 2023-12-06T03:55:17.818 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.816+0000 7f2e3412c700 1 -- 172.21.15.152:0/309493377 shutdown_connections 2023-12-06T03:55:17.818 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.816+0000 7f2e3412c700 1 --2- 172.21.15.152:0/309493377 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2e1802ff60 0x7f2e18032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:17.818 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.816+0000 7f2e3412c700 1 --2- 172.21.15.152:0/309493377 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2e2c0f6510 0x7f2e2c106730 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:17.818 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.816+0000 7f2e3412c700 1 -- 172.21.15.152:0/309493377 >> 172.21.15.152:0/309493377 conn(0x7f2e2c0f1c70 msgr2=0x7f2e2c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:17.818 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.816+0000 7f2e3412c700 1 -- 172.21.15.152:0/309493377 shutdown_connections 2023-12-06T03:55:17.818 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:17.816+0000 7f2e3412c700 1 -- 172.21.15.152:0/309493377 wait complete. 2023-12-06T03:55:17.818 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:18.110 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:17 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:16.403618+0000 mgr.smithi132.gokxja (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:19.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:18 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:17.815481+0000 mon.smithi132 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.152:0/309493377' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:19.184 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:19.184 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:20.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:19 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:18.404040+0000 mgr.smithi132.gokxja (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:20.307 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.304+0000 7f7738a0c700 1 -- 172.21.15.152:0/2830352926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77340f5410 msgr2=0x7f77340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:20.307 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.304+0000 7f7738a0c700 1 --2- 172.21.15.152:0/2830352926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77340f5410 0x7f77340f5830 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f7728009fb0 tx=0x7f7728009b30).stop 2023-12-06T03:55:20.307 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.304+0000 7f7738a0c700 1 -- 172.21.15.152:0/2830352926 shutdown_connections 2023-12-06T03:55:20.307 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.304+0000 7f7738a0c700 1 --2- 172.21.15.152:0/2830352926 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77340f5410 0x7f77340f5830 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:20.307 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.304+0000 7f7738a0c700 1 -- 172.21.15.152:0/2830352926 >> 172.21.15.152:0/2830352926 conn(0x7f77340f0b70 msgr2=0x7f77340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:20.307 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.304+0000 7f7738a0c700 1 -- 172.21.15.152:0/2830352926 shutdown_connections 2023-12-06T03:55:20.307 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.304+0000 7f7738a0c700 1 -- 172.21.15.152:0/2830352926 wait complete. 2023-12-06T03:55:20.307 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.305+0000 7f7738a0c700 1 Processor -- start 2023-12-06T03:55:20.307 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.305+0000 7f7738a0c700 1 -- start start 2023-12-06T03:55:20.307 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.305+0000 7f7738a0c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77340f5410 0x7f773410b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:20.307 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.305+0000 7f7738a0c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77341071e0 con 0x7f77340f5410 2023-12-06T03:55:20.309 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.305+0000 7f773259c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77340f5410 0x7f773410b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:20.309 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.305+0000 7f773259c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77340f5410 0x7f773410b7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40462/0 (socket says 172.21.15.152:40462) 2023-12-06T03:55:20.309 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.305+0000 7f773259c700 1 -- 172.21.15.152:0/1140405013 learned_addr learned my addr 172.21.15.152:0/1140405013 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:20.309 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.306+0000 7f773259c700 1 -- 172.21.15.152:0/1140405013 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7728005040 con 0x7f77340f5410 2023-12-06T03:55:20.309 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.306+0000 7f773259c700 1 --2- 172.21.15.152:0/1140405013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77340f5410 0x7f773410b7f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f7728013510 tx=0x7f772800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:20.310 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.306+0000 7f7722ffd700 1 -- 172.21.15.152:0/1140405013 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7728009320 con 0x7f77340f5410 2023-12-06T03:55:20.310 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.306+0000 7f7722ffd700 1 -- 172.21.15.152:0/1140405013 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7728013f00 con 0x7f77340f5410 2023-12-06T03:55:20.310 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.306+0000 7f7738a0c700 1 -- 172.21.15.152:0/1140405013 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7734060220 con 0x7f77340f5410 2023-12-06T03:55:20.310 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.306+0000 7f7722ffd700 1 -- 172.21.15.152:0/1140405013 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7728014d90 con 0x7f77340f5410 2023-12-06T03:55:20.310 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.306+0000 7f7738a0c700 1 -- 172.21.15.152:0/1140405013 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f773410bce0 con 0x7f77340f5410 2023-12-06T03:55:20.310 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.307+0000 7f7722ffd700 1 -- 172.21.15.152:0/1140405013 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f77280095f0 con 0x7f77340f5410 2023-12-06T03:55:20.310 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.307+0000 7f7738a0c700 1 -- 172.21.15.152:0/1140405013 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7734045ad0 con 0x7f77340f5410 2023-12-06T03:55:20.310 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.307+0000 7f7722ffd700 1 --2- 172.21.15.152:0/1140405013 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f771c02ff60 0x7f771c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:20.310 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.307+0000 7f7722ffd700 1 -- 172.21.15.152:0/1140405013 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f77280410c0 con 0x7f77340f5410 2023-12-06T03:55:20.310 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.307+0000 7f7731d9b700 1 --2- 172.21.15.152:0/1140405013 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f771c02ff60 0x7f771c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:20.310 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.308+0000 7f7731d9b700 1 --2- 172.21.15.152:0/1140405013 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f771c02ff60 0x7f771c032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f77240096a0 tx=0x7f7724006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:20.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.312+0000 7f7722ffd700 1 -- 172.21.15.152:0/1140405013 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f772801d7f0 con 0x7f77340f5410 2023-12-06T03:55:20.521 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.519+0000 7f7738a0c700 1 -- 172.21.15.152:0/1140405013 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f773410c4a0 con 0x7f77340f5410 2023-12-06T03:55:20.522 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.520+0000 7f7722ffd700 1 -- 172.21.15.152:0/1140405013 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7728010230 con 0x7f77340f5410 2023-12-06T03:55:20.522 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:20.522 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:20.524 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.522+0000 7f7738a0c700 1 -- 172.21.15.152:0/1140405013 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f771c02ff60 msgr2=0x7f771c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:20.524 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.522+0000 7f7738a0c700 1 --2- 172.21.15.152:0/1140405013 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f771c02ff60 0x7f771c032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f77240096a0 tx=0x7f7724006b40).stop 2023-12-06T03:55:20.524 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.522+0000 7f7738a0c700 1 -- 172.21.15.152:0/1140405013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77340f5410 msgr2=0x7f773410b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:20.524 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.522+0000 7f7738a0c700 1 --2- 172.21.15.152:0/1140405013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77340f5410 0x7f773410b7f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f7728013510 tx=0x7f772800b040).stop 2023-12-06T03:55:20.524 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.522+0000 7f7738a0c700 1 -- 172.21.15.152:0/1140405013 shutdown_connections 2023-12-06T03:55:20.524 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.522+0000 7f7738a0c700 1 --2- 172.21.15.152:0/1140405013 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f771c02ff60 0x7f771c032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:20.524 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.522+0000 7f7738a0c700 1 --2- 172.21.15.152:0/1140405013 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f77340f5410 0x7f773410b7f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:20.524 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.522+0000 7f7738a0c700 1 -- 172.21.15.152:0/1140405013 >> 172.21.15.152:0/1140405013 conn(0x7f77340f0b70 msgr2=0x7f77340f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:20.524 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.522+0000 7f7738a0c700 1 -- 172.21.15.152:0/1140405013 shutdown_connections 2023-12-06T03:55:20.524 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:20.522+0000 7f7738a0c700 1 -- 172.21.15.152:0/1140405013 wait complete. 2023-12-06T03:55:20.524 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:21.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:20.521218+0000 mon.smithi132 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.152:0/1140405013' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:21.902 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:21.902 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:22.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:20.404383+0000 mgr.smithi132.gokxja (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:22.986 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.984+0000 7fd2184d5700 1 -- 172.21.15.152:0/624339004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210060280 msgr2=0x7fd2100f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:22.987 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.984+0000 7fd2184d5700 1 --2- 172.21.15.152:0/624339004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210060280 0x7fd2100f90f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fd200009fb0 tx=0x7fd200009b30).stop 2023-12-06T03:55:22.987 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.984+0000 7fd2184d5700 1 -- 172.21.15.152:0/624339004 shutdown_connections 2023-12-06T03:55:22.987 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.984+0000 7fd2184d5700 1 --2- 172.21.15.152:0/624339004 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210060280 0x7fd2100f90f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:22.987 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.984+0000 7fd2184d5700 1 -- 172.21.15.152:0/624339004 >> 172.21.15.152:0/624339004 conn(0x7fd2100f0b70 msgr2=0x7fd2100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:22.987 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.984+0000 7fd2184d5700 1 -- 172.21.15.152:0/624339004 shutdown_connections 2023-12-06T03:55:22.987 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.984+0000 7fd2184d5700 1 -- 172.21.15.152:0/624339004 wait complete. 2023-12-06T03:55:22.987 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.985+0000 7fd2184d5700 1 Processor -- start 2023-12-06T03:55:22.987 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.985+0000 7fd2184d5700 1 -- start start 2023-12-06T03:55:22.987 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.985+0000 7fd2184d5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210060280 0x7fd2101066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:22.987 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.985+0000 7fd2184d5700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd210106bd0 con 0x7fd210060280 2023-12-06T03:55:22.989 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.985+0000 7fd216271700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210060280 0x7fd2101066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:22.989 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.985+0000 7fd216271700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210060280 0x7fd2101066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40476/0 (socket says 172.21.15.152:40476) 2023-12-06T03:55:22.989 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.985+0000 7fd216271700 1 -- 172.21.15.152:0/2618500921 learned_addr learned my addr 172.21.15.152:0/2618500921 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:22.989 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.985+0000 7fd216271700 1 -- 172.21.15.152:0/2618500921 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd200005040 con 0x7fd210060280 2023-12-06T03:55:22.989 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.986+0000 7fd216271700 1 --2- 172.21.15.152:0/2618500921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210060280 0x7fd2101066e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd200013510 tx=0x7fd20000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:22.989 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.986+0000 7fd206ffd700 1 -- 172.21.15.152:0/2618500921 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd200009320 con 0x7fd210060280 2023-12-06T03:55:22.989 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.986+0000 7fd2184d5700 1 -- 172.21.15.152:0/2618500921 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd210106dd0 con 0x7fd210060280 2023-12-06T03:55:22.989 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.986+0000 7fd206ffd700 1 -- 172.21.15.152:0/2618500921 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd200013f00 con 0x7fd210060280 2023-12-06T03:55:22.989 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.986+0000 7fd206ffd700 1 -- 172.21.15.152:0/2618500921 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd200014dc0 con 0x7fd210060280 2023-12-06T03:55:22.989 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.986+0000 7fd2184d5700 1 -- 172.21.15.152:0/2618500921 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd210107260 con 0x7fd210060280 2023-12-06T03:55:22.989 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.986+0000 7fd206ffd700 1 -- 172.21.15.152:0/2618500921 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd200014420 con 0x7fd210060280 2023-12-06T03:55:22.989 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.987+0000 7fd2184d5700 1 -- 172.21.15.152:0/2618500921 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd210045ad0 con 0x7fd210060280 2023-12-06T03:55:22.990 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.987+0000 7fd206ffd700 1 --2- 172.21.15.152:0/2618500921 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd1fc02ff60 0x7fd1fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:22.990 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.987+0000 7fd206ffd700 1 -- 172.21.15.152:0/2618500921 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd200041110 con 0x7fd210060280 2023-12-06T03:55:22.990 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.987+0000 7fd215a70700 1 --2- 172.21.15.152:0/2618500921 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd1fc02ff60 0x7fd1fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:22.990 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.987+0000 7fd215a70700 1 --2- 172.21.15.152:0/2618500921 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd1fc02ff60 0x7fd1fc032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fd20c0096a0 tx=0x7fd20c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:22.993 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:22.991+0000 7fd206ffd700 1 -- 172.21.15.152:0/2618500921 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd2000095f0 con 0x7fd210060280 2023-12-06T03:55:23.200 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:23.198+0000 7fd2184d5700 1 -- 172.21.15.152:0/2618500921 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd210108110 con 0x7fd210060280 2023-12-06T03:55:23.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:23.199+0000 7fd206ffd700 1 -- 172.21.15.152:0/2618500921 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd20001cb10 con 0x7fd210060280 2023-12-06T03:55:23.201 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:23.201 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:23.202 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:23.201+0000 7fd2184d5700 1 -- 172.21.15.152:0/2618500921 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd1fc02ff60 msgr2=0x7fd1fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:23.203 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:23.201+0000 7fd2184d5700 1 --2- 172.21.15.152:0/2618500921 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd1fc02ff60 0x7fd1fc032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fd20c0096a0 tx=0x7fd20c006b40).stop 2023-12-06T03:55:23.203 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:23.201+0000 7fd2184d5700 1 -- 172.21.15.152:0/2618500921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210060280 msgr2=0x7fd2101066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:23.203 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:23.201+0000 7fd2184d5700 1 --2- 172.21.15.152:0/2618500921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210060280 0x7fd2101066e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd200013510 tx=0x7fd20000b040).stop 2023-12-06T03:55:23.203 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:23.201+0000 7fd2184d5700 1 -- 172.21.15.152:0/2618500921 shutdown_connections 2023-12-06T03:55:23.203 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:23.201+0000 7fd2184d5700 1 --2- 172.21.15.152:0/2618500921 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd1fc02ff60 0x7fd1fc032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:23.203 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:23.201+0000 7fd2184d5700 1 --2- 172.21.15.152:0/2618500921 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd210060280 0x7fd2101066e0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:23.203 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:23.201+0000 7fd2184d5700 1 -- 172.21.15.152:0/2618500921 >> 172.21.15.152:0/2618500921 conn(0x7fd2100f0b70 msgr2=0x7fd2100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:23.203 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:23.201+0000 7fd2184d5700 1 -- 172.21.15.152:0/2618500921 shutdown_connections 2023-12-06T03:55:23.203 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:23.201+0000 7fd2184d5700 1 -- 172.21.15.152:0/2618500921 wait complete. 2023-12-06T03:55:23.203 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:24.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:22.404771+0000 mgr.smithi132.gokxja (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:24.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:23.200342+0000 mon.smithi132 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.152:0/2618500921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:24.567 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:24.567 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:25.683 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.678+0000 7f0618a22700 1 -- 172.21.15.152:0/17429126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06140f5410 msgr2=0x7f06140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:25.683 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.678+0000 7f0618a22700 1 --2- 172.21.15.152:0/17429126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06140f5410 0x7f06140f5830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f0608009fb0 tx=0x7f0608009b30).stop 2023-12-06T03:55:25.684 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.678+0000 7f0618a22700 1 -- 172.21.15.152:0/17429126 shutdown_connections 2023-12-06T03:55:25.684 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.678+0000 7f0618a22700 1 --2- 172.21.15.152:0/17429126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06140f5410 0x7f06140f5830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:25.684 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.678+0000 7f0618a22700 1 -- 172.21.15.152:0/17429126 >> 172.21.15.152:0/17429126 conn(0x7f06140f0b70 msgr2=0x7f06140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:25.684 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.678+0000 7f0618a22700 1 -- 172.21.15.152:0/17429126 shutdown_connections 2023-12-06T03:55:25.684 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.678+0000 7f0618a22700 1 -- 172.21.15.152:0/17429126 wait complete. 2023-12-06T03:55:25.684 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.679+0000 7f0618a22700 1 Processor -- start 2023-12-06T03:55:25.684 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.679+0000 7f0618a22700 1 -- start start 2023-12-06T03:55:25.684 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.679+0000 7f0618a22700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06140f5410 0x7f0614105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:25.684 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.679+0000 7f0618a22700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0614105b20 con 0x7f06140f5410 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.679+0000 7f061259c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06140f5410 0x7f0614105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.679+0000 7f061259c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06140f5410 0x7f0614105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40498/0 (socket says 172.21.15.152:40498) 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.679+0000 7f061259c700 1 -- 172.21.15.152:0/2708741871 learned_addr learned my addr 172.21.15.152:0/2708741871 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.679+0000 7f061259c700 1 -- 172.21.15.152:0/2708741871 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0608005040 con 0x7f06140f5410 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.679+0000 7f061259c700 1 --2- 172.21.15.152:0/2708741871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06140f5410 0x7f0614105630 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0608014000 tx=0x7f060800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.679+0000 7f0602ffd700 1 -- 172.21.15.152:0/2708741871 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06080095f0 con 0x7f06140f5410 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.680+0000 7f0618a22700 1 -- 172.21.15.152:0/2708741871 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0614105d20 con 0x7f06140f5410 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.680+0000 7f0602ffd700 1 -- 172.21.15.152:0/2708741871 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0608007e70 con 0x7f06140f5410 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.680+0000 7f0602ffd700 1 -- 172.21.15.152:0/2708741871 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0608014ea0 con 0x7f06140f5410 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.680+0000 7f0618a22700 1 -- 172.21.15.152:0/2708741871 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06141061b0 con 0x7f06140f5410 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.680+0000 7f0602ffd700 1 -- 172.21.15.152:0/2708741871 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06080079c0 con 0x7f06140f5410 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.680+0000 7f0618a22700 1 -- 172.21.15.152:0/2708741871 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0614045ad0 con 0x7f06140f5410 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.680+0000 7f0602ffd700 1 --2- 172.21.15.152:0/2708741871 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f05fc02ff60 0x7f05fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.681+0000 7f0602ffd700 1 -- 172.21.15.152:0/2708741871 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0608040900 con 0x7f06140f5410 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.681+0000 7f0611d9b700 1 --2- 172.21.15.152:0/2708741871 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f05fc02ff60 0x7f05fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:25.685 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.681+0000 7f0611d9b700 1 --2- 172.21.15.152:0/2708741871 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f05fc02ff60 0x7f05fc032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f06040096a0 tx=0x7f0604006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:25.687 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.685+0000 7f0602ffd700 1 -- 172.21.15.152:0/2708741871 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f060801da20 con 0x7f06140f5410 2023-12-06T03:55:25.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.892+0000 7f0618a22700 1 -- 172.21.15.152:0/2708741871 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06140fae60 con 0x7f06140f5410 2023-12-06T03:55:25.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.893+0000 7f0602ffd700 1 -- 172.21.15.152:0/2708741871 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0608010230 con 0x7f06140f5410 2023-12-06T03:55:25.895 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:25.895 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:25.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.894+0000 7f0618a22700 1 -- 172.21.15.152:0/2708741871 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f05fc02ff60 msgr2=0x7f05fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:25.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.894+0000 7f0618a22700 1 --2- 172.21.15.152:0/2708741871 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f05fc02ff60 0x7f05fc032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f06040096a0 tx=0x7f0604006b40).stop 2023-12-06T03:55:25.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.894+0000 7f0618a22700 1 -- 172.21.15.152:0/2708741871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06140f5410 msgr2=0x7f0614105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:25.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.894+0000 7f0618a22700 1 --2- 172.21.15.152:0/2708741871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06140f5410 0x7f0614105630 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0608014000 tx=0x7f060800b040).stop 2023-12-06T03:55:25.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.895+0000 7f0618a22700 1 -- 172.21.15.152:0/2708741871 shutdown_connections 2023-12-06T03:55:25.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.895+0000 7f0618a22700 1 --2- 172.21.15.152:0/2708741871 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f05fc02ff60 0x7f05fc032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:25.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.895+0000 7f0618a22700 1 --2- 172.21.15.152:0/2708741871 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f06140f5410 0x7f0614105630 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:25.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.895+0000 7f0618a22700 1 -- 172.21.15.152:0/2708741871 >> 172.21.15.152:0/2708741871 conn(0x7f06140f0b70 msgr2=0x7f06140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:25.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.895+0000 7f0618a22700 1 -- 172.21.15.152:0/2708741871 shutdown_connections 2023-12-06T03:55:25.897 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:25.895+0000 7f0618a22700 1 -- 172.21.15.152:0/2708741871 wait complete. 2023-12-06T03:55:25.897 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:26.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:24.405214+0000 mgr.smithi132.gokxja (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:27.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:26 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:25.894299+0000 mon.smithi132 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.152:0/2708741871' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:27.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:27.270 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:28.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:26.405608+0000 mgr.smithi132.gokxja (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:28.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.358+0000 7f53ffa7a700 1 -- 172.21.15.152:0/250984810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53f80f6510 msgr2=0x7f53f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:28.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.358+0000 7f53ffa7a700 1 --2- 172.21.15.152:0/250984810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53f80f6510 0x7f53f80f6930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f53e8009fb0 tx=0x7f53e8009b30).stop 2023-12-06T03:55:28.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.358+0000 7f53ffa7a700 1 -- 172.21.15.152:0/250984810 shutdown_connections 2023-12-06T03:55:28.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.358+0000 7f53ffa7a700 1 --2- 172.21.15.152:0/250984810 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53f80f6510 0x7f53f80f6930 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:28.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.358+0000 7f53ffa7a700 1 -- 172.21.15.152:0/250984810 >> 172.21.15.152:0/250984810 conn(0x7f53f80f1c70 msgr2=0x7f53f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:28.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.358+0000 7f53ffa7a700 1 -- 172.21.15.152:0/250984810 shutdown_connections 2023-12-06T03:55:28.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.358+0000 7f53ffa7a700 1 -- 172.21.15.152:0/250984810 wait complete. 2023-12-06T03:55:28.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.359+0000 7f53ffa7a700 1 Processor -- start 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.359+0000 7f53ffa7a700 1 -- start start 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.359+0000 7f53ffa7a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53f80f6510 0x7f53f8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.359+0000 7f53ffa7a700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f8106c20 con 0x7f53f80f6510 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.359+0000 7f53fd816700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53f80f6510 0x7f53f8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.359+0000 7f53fd816700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53f80f6510 0x7f53f8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40514/0 (socket says 172.21.15.152:40514) 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.359+0000 7f53fd816700 1 -- 172.21.15.152:0/1390784913 learned_addr learned my addr 172.21.15.152:0/1390784913 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.360+0000 7f53fd816700 1 -- 172.21.15.152:0/1390784913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e8005040 con 0x7f53f80f6510 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.360+0000 7f53fd816700 1 --2- 172.21.15.152:0/1390784913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53f80f6510 0x7f53f8106730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f53e8013510 tx=0x7f53e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.360+0000 7f53ee7fc700 1 -- 172.21.15.152:0/1390784913 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53e8009320 con 0x7f53f80f6510 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.360+0000 7f53ffa7a700 1 -- 172.21.15.152:0/1390784913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f8106e20 con 0x7f53f80f6510 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.360+0000 7f53ee7fc700 1 -- 172.21.15.152:0/1390784913 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53e8013f00 con 0x7f53f80f6510 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.360+0000 7f53ee7fc700 1 -- 172.21.15.152:0/1390784913 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53e8014d90 con 0x7f53f80f6510 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.360+0000 7f53ffa7a700 1 -- 172.21.15.152:0/1390784913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f81072b0 con 0x7f53f80f6510 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.361+0000 7f53ee7fc700 1 -- 172.21.15.152:0/1390784913 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53e8014420 con 0x7f53f80f6510 2023-12-06T03:55:28.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.361+0000 7f53ffa7a700 1 -- 172.21.15.152:0/1390784913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f8045ad0 con 0x7f53f80f6510 2023-12-06T03:55:28.365 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.361+0000 7f53ee7fc700 1 --2- 172.21.15.152:0/1390784913 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f53e402ff60 0x7f53e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:28.365 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.361+0000 7f53ee7fc700 1 -- 172.21.15.152:0/1390784913 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f53e8041000 con 0x7f53f80f6510 2023-12-06T03:55:28.365 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.361+0000 7f53fd015700 1 --2- 172.21.15.152:0/1390784913 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f53e402ff60 0x7f53e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:28.365 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.362+0000 7f53fd015700 1 --2- 172.21.15.152:0/1390784913 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f53e402ff60 0x7f53e4032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f53f40096a0 tx=0x7f53f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:28.367 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.366+0000 7f53ee7fc700 1 -- 172.21.15.152:0/1390784913 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53e80095f0 con 0x7f53f80f6510 2023-12-06T03:55:28.575 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.573+0000 7f53ffa7a700 1 -- 172.21.15.152:0/1390784913 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53f80fbf60 con 0x7f53f80f6510 2023-12-06T03:55:28.576 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.574+0000 7f53ee7fc700 1 -- 172.21.15.152:0/1390784913 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f53e801db70 con 0x7f53f80f6510 2023-12-06T03:55:28.576 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:28.576 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:28.577 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.576+0000 7f53ffa7a700 1 -- 172.21.15.152:0/1390784913 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f53e402ff60 msgr2=0x7f53e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:28.578 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.576+0000 7f53ffa7a700 1 --2- 172.21.15.152:0/1390784913 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f53e402ff60 0x7f53e4032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f53f40096a0 tx=0x7f53f4006b40).stop 2023-12-06T03:55:28.578 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.576+0000 7f53ffa7a700 1 -- 172.21.15.152:0/1390784913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53f80f6510 msgr2=0x7f53f8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:28.578 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.576+0000 7f53ffa7a700 1 --2- 172.21.15.152:0/1390784913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53f80f6510 0x7f53f8106730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f53e8013510 tx=0x7f53e800b040).stop 2023-12-06T03:55:28.578 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.576+0000 7f53ffa7a700 1 -- 172.21.15.152:0/1390784913 shutdown_connections 2023-12-06T03:55:28.578 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.576+0000 7f53ffa7a700 1 --2- 172.21.15.152:0/1390784913 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f53e402ff60 0x7f53e4032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:28.578 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.576+0000 7f53ffa7a700 1 --2- 172.21.15.152:0/1390784913 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f53f80f6510 0x7f53f8106730 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:28.578 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.576+0000 7f53ffa7a700 1 -- 172.21.15.152:0/1390784913 >> 172.21.15.152:0/1390784913 conn(0x7f53f80f1c70 msgr2=0x7f53f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:28.578 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.576+0000 7f53ffa7a700 1 -- 172.21.15.152:0/1390784913 shutdown_connections 2023-12-06T03:55:28.578 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:28.576+0000 7f53ffa7a700 1 -- 172.21.15.152:0/1390784913 wait complete. 2023-12-06T03:55:28.578 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:29.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:28 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:28.575389+0000 mon.smithi132 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.152:0/1390784913' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:29.937 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:29.938 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:30.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:28.405910+0000 mgr.smithi132.gokxja (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:31.004 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:30.999+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/3322469627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9eb80f6510 msgr2=0x7f9eb80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:30.999+0000 7f9ebfdfa700 1 --2- 172.21.15.152:0/3322469627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9eb80f6510 0x7f9eb80f6930 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f9ea8009fb0 tx=0x7f9ea8009b30).stop 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:30.999+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/3322469627 shutdown_connections 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:30.999+0000 7f9ebfdfa700 1 --2- 172.21.15.152:0/3322469627 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9eb80f6510 0x7f9eb80f6930 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:30.999+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/3322469627 >> 172.21.15.152:0/3322469627 conn(0x7f9eb80f1c70 msgr2=0x7f9eb80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:30.999+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/3322469627 shutdown_connections 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:30.999+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/3322469627 wait complete. 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:30.999+0000 7f9ebfdfa700 1 Processor -- start 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.000+0000 7f9ebfdfa700 1 -- start start 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.000+0000 7f9ebfdfa700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9eb80f6510 0x7f9eb8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.000+0000 7f9ebfdfa700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eb8106c20 con 0x7f9eb80f6510 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.000+0000 7f9ebdb96700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9eb80f6510 0x7f9eb8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.000+0000 7f9ebdb96700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9eb80f6510 0x7f9eb8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:49764/0 (socket says 172.21.15.152:49764) 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.000+0000 7f9ebdb96700 1 -- 172.21.15.152:0/1676825300 learned_addr learned my addr 172.21.15.152:0/1676825300 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.000+0000 7f9ebdb96700 1 -- 172.21.15.152:0/1676825300 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ea8005040 con 0x7f9eb80f6510 2023-12-06T03:55:31.005 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.000+0000 7f9ebdb96700 1 --2- 172.21.15.152:0/1676825300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9eb80f6510 0x7f9eb8106730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9ea8013510 tx=0x7f9ea800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:31.006 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.001+0000 7f9eae7fc700 1 -- 172.21.15.152:0/1676825300 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ea8009320 con 0x7f9eb80f6510 2023-12-06T03:55:31.006 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.001+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/1676825300 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb8106e20 con 0x7f9eb80f6510 2023-12-06T03:55:31.006 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.001+0000 7f9eae7fc700 1 -- 172.21.15.152:0/1676825300 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ea8013f00 con 0x7f9eb80f6510 2023-12-06T03:55:31.006 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.001+0000 7f9eae7fc700 1 -- 172.21.15.152:0/1676825300 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ea8014d90 con 0x7f9eb80f6510 2023-12-06T03:55:31.006 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.001+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/1676825300 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb81072b0 con 0x7f9eb80f6510 2023-12-06T03:55:31.006 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.001+0000 7f9eae7fc700 1 -- 172.21.15.152:0/1676825300 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ea80095f0 con 0x7f9eb80f6510 2023-12-06T03:55:31.006 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.001+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/1676825300 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eb8045ad0 con 0x7f9eb80f6510 2023-12-06T03:55:31.006 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.002+0000 7f9eae7fc700 1 --2- 172.21.15.152:0/1676825300 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9ea402ff60 0x7f9ea4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:31.006 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.002+0000 7f9eae7fc700 1 -- 172.21.15.152:0/1676825300 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ea80410c0 con 0x7f9eb80f6510 2023-12-06T03:55:31.006 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.002+0000 7f9ebd395700 1 --2- 172.21.15.152:0/1676825300 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9ea402ff60 0x7f9ea4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:31.006 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.002+0000 7f9ebd395700 1 --2- 172.21.15.152:0/1676825300 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9ea402ff60 0x7f9ea4032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9eb40096a0 tx=0x7f9eb4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:31.008 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.006+0000 7f9eae7fc700 1 -- 172.21.15.152:0/1676825300 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ea8014420 con 0x7f9eb80f6510 2023-12-06T03:55:31.215 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.213+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/1676825300 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9eb80fbf60 con 0x7f9eb80f6510 2023-12-06T03:55:31.216 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.214+0000 7f9eae7fc700 1 -- 172.21.15.152:0/1676825300 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9ea801d7f0 con 0x7f9eb80f6510 2023-12-06T03:55:31.216 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:31.216 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.216+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/1676825300 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9ea402ff60 msgr2=0x7f9ea4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.216+0000 7f9ebfdfa700 1 --2- 172.21.15.152:0/1676825300 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9ea402ff60 0x7f9ea4032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9eb40096a0 tx=0x7f9eb4006b40).stop 2023-12-06T03:55:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.216+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/1676825300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9eb80f6510 msgr2=0x7f9eb8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.216+0000 7f9ebfdfa700 1 --2- 172.21.15.152:0/1676825300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9eb80f6510 0x7f9eb8106730 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9ea8013510 tx=0x7f9ea800b040).stop 2023-12-06T03:55:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.216+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/1676825300 shutdown_connections 2023-12-06T03:55:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.216+0000 7f9ebfdfa700 1 --2- 172.21.15.152:0/1676825300 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9ea402ff60 0x7f9ea4032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.216+0000 7f9ebfdfa700 1 --2- 172.21.15.152:0/1676825300 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9eb80f6510 0x7f9eb8106730 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.216+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/1676825300 >> 172.21.15.152:0/1676825300 conn(0x7f9eb80f1c70 msgr2=0x7f9eb80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.216+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/1676825300 shutdown_connections 2023-12-06T03:55:31.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:31.216+0000 7f9ebfdfa700 1 -- 172.21.15.152:0/1676825300 wait complete. 2023-12-06T03:55:31.219 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:32.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:31 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:30.406275+0000 mgr.smithi132.gokxja (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:32.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:31 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:31.215563+0000 mon.smithi132 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.152:0/1676825300' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:32.596 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:32.597 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:33.726 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.724+0000 7f279affc700 1 -- 172.21.15.152:0/2307038320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27940f5410 msgr2=0x7f27940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:33.726 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.724+0000 7f279affc700 1 --2- 172.21.15.152:0/2307038320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27940f5410 0x7f27940f5830 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f2788009fb0 tx=0x7f2788009b30).stop 2023-12-06T03:55:33.726 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.724+0000 7f279affc700 1 -- 172.21.15.152:0/2307038320 shutdown_connections 2023-12-06T03:55:33.726 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.724+0000 7f279affc700 1 --2- 172.21.15.152:0/2307038320 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27940f5410 0x7f27940f5830 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:33.726 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.724+0000 7f279affc700 1 -- 172.21.15.152:0/2307038320 >> 172.21.15.152:0/2307038320 conn(0x7f27940f0b70 msgr2=0x7f27940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:33.726 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.724+0000 7f279affc700 1 -- 172.21.15.152:0/2307038320 shutdown_connections 2023-12-06T03:55:33.727 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.724+0000 7f279affc700 1 -- 172.21.15.152:0/2307038320 wait complete. 2023-12-06T03:55:33.727 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.724+0000 7f279affc700 1 Processor -- start 2023-12-06T03:55:33.727 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.724+0000 7f279affc700 1 -- start start 2023-12-06T03:55:33.728 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.725+0000 7f279affc700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27940f5410 0x7f2794105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:33.728 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.725+0000 7f279affc700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2794105b20 con 0x7f27940f5410 2023-12-06T03:55:33.728 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.725+0000 7f2798d98700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27940f5410 0x7f2794105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:33.728 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.725+0000 7f2798d98700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27940f5410 0x7f2794105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:49784/0 (socket says 172.21.15.152:49784) 2023-12-06T03:55:33.728 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.725+0000 7f2798d98700 1 -- 172.21.15.152:0/2477785114 learned_addr learned my addr 172.21.15.152:0/2477785114 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:33.728 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.725+0000 7f2798d98700 1 -- 172.21.15.152:0/2477785114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2788005040 con 0x7f27940f5410 2023-12-06T03:55:33.729 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.725+0000 7f2798d98700 1 --2- 172.21.15.152:0/2477785114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27940f5410 0x7f2794105630 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f2788013510 tx=0x7f278800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:33.729 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.725+0000 7f27917fa700 1 -- 172.21.15.152:0/2477785114 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2788009320 con 0x7f27940f5410 2023-12-06T03:55:33.729 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.725+0000 7f279affc700 1 -- 172.21.15.152:0/2477785114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2794105d20 con 0x7f27940f5410 2023-12-06T03:55:33.729 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.726+0000 7f27917fa700 1 -- 172.21.15.152:0/2477785114 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2788013f00 con 0x7f27940f5410 2023-12-06T03:55:33.729 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.726+0000 7f27917fa700 1 -- 172.21.15.152:0/2477785114 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2788014d90 con 0x7f27940f5410 2023-12-06T03:55:33.729 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.726+0000 7f279affc700 1 -- 172.21.15.152:0/2477785114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27941061b0 con 0x7f27940f5410 2023-12-06T03:55:33.729 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.726+0000 7f27917fa700 1 -- 172.21.15.152:0/2477785114 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f27880095f0 con 0x7f27940f5410 2023-12-06T03:55:33.729 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.726+0000 7f279affc700 1 -- 172.21.15.152:0/2477785114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2794045ad0 con 0x7f27940f5410 2023-12-06T03:55:33.729 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.726+0000 7f27917fa700 1 --2- 172.21.15.152:0/2477785114 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f278402ff60 0x7f2784032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:33.729 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.727+0000 7f27917fa700 1 -- 172.21.15.152:0/2477785114 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2788041080 con 0x7f27940f5410 2023-12-06T03:55:33.729 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.727+0000 7f2793fff700 1 --2- 172.21.15.152:0/2477785114 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f278402ff60 0x7f2784032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:33.729 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.727+0000 7f2793fff700 1 --2- 172.21.15.152:0/2477785114 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f278402ff60 0x7f2784032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f27800096a0 tx=0x7f2780006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:33.733 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.731+0000 7f27917fa700 1 -- 172.21.15.152:0/2477785114 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2788014420 con 0x7f27940f5410 2023-12-06T03:55:33.939 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.937+0000 7f279affc700 1 -- 172.21.15.152:0/2477785114 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f27940fae60 con 0x7f27940f5410 2023-12-06T03:55:33.939 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.938+0000 7f27917fa700 1 -- 172.21.15.152:0/2477785114 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2788010230 con 0x7f27940f5410 2023-12-06T03:55:33.940 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:33.940 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:33.942 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.940+0000 7f279affc700 1 -- 172.21.15.152:0/2477785114 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f278402ff60 msgr2=0x7f2784032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:33.942 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.940+0000 7f279affc700 1 --2- 172.21.15.152:0/2477785114 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f278402ff60 0x7f2784032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f27800096a0 tx=0x7f2780006b40).stop 2023-12-06T03:55:33.942 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.940+0000 7f279affc700 1 -- 172.21.15.152:0/2477785114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27940f5410 msgr2=0x7f2794105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:33.942 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.940+0000 7f279affc700 1 --2- 172.21.15.152:0/2477785114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27940f5410 0x7f2794105630 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f2788013510 tx=0x7f278800b040).stop 2023-12-06T03:55:33.942 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.940+0000 7f279affc700 1 -- 172.21.15.152:0/2477785114 shutdown_connections 2023-12-06T03:55:33.942 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.940+0000 7f279affc700 1 --2- 172.21.15.152:0/2477785114 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f278402ff60 0x7f2784032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:33.942 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.940+0000 7f279affc700 1 --2- 172.21.15.152:0/2477785114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f27940f5410 0x7f2794105630 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:33.942 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.940+0000 7f279affc700 1 -- 172.21.15.152:0/2477785114 >> 172.21.15.152:0/2477785114 conn(0x7f27940f0b70 msgr2=0x7f27940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:33.942 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.940+0000 7f279affc700 1 -- 172.21.15.152:0/2477785114 shutdown_connections 2023-12-06T03:55:33.943 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:33.940+0000 7f279affc700 1 -- 172.21.15.152:0/2477785114 wait complete. 2023-12-06T03:55:33.943 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:34.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:33 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:32.406646+0000 mgr.smithi132.gokxja (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:35.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:34 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:33.939363+0000 mon.smithi132 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.152:0/2477785114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:35.328 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:35.328 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:36.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:35 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:34.407004+0000 mgr.smithi132.gokxja (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:36.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.447+0000 7f83377b9700 1 -- 172.21.15.152:0/3178142610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83300f47e0 msgr2=0x7f83300f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:36.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.447+0000 7f83377b9700 1 --2- 172.21.15.152:0/3178142610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83300f47e0 0x7f83300f4c00 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f8320009fb0 tx=0x7f8320009b30).stop 2023-12-06T03:55:36.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.447+0000 7f83377b9700 1 -- 172.21.15.152:0/3178142610 shutdown_connections 2023-12-06T03:55:36.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.447+0000 7f83377b9700 1 --2- 172.21.15.152:0/3178142610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83300f47e0 0x7f83300f4c00 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:36.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.447+0000 7f83377b9700 1 -- 172.21.15.152:0/3178142610 >> 172.21.15.152:0/3178142610 conn(0x7f83300f0260 msgr2=0x7f83300f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:36.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.447+0000 7f83377b9700 1 -- 172.21.15.152:0/3178142610 shutdown_connections 2023-12-06T03:55:36.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.447+0000 7f83377b9700 1 -- 172.21.15.152:0/3178142610 wait complete. 2023-12-06T03:55:36.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.448+0000 7f83377b9700 1 Processor -- start 2023-12-06T03:55:36.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.448+0000 7f83377b9700 1 -- start start 2023-12-06T03:55:36.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.448+0000 7f83377b9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83300f47e0 0x7f83301068d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:36.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.448+0000 7f83377b9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8330106dc0 con 0x7f83300f47e0 2023-12-06T03:55:36.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.448+0000 7f8335555700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83300f47e0 0x7f83301068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:36.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.448+0000 7f8335555700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83300f47e0 0x7f83301068d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:49810/0 (socket says 172.21.15.152:49810) 2023-12-06T03:55:36.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.448+0000 7f8335555700 1 -- 172.21.15.152:0/3792394396 learned_addr learned my addr 172.21.15.152:0/3792394396 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:36.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.448+0000 7f8335555700 1 -- 172.21.15.152:0/3792394396 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8320005040 con 0x7f83300f47e0 2023-12-06T03:55:36.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.448+0000 7f8335555700 1 --2- 172.21.15.152:0/3792394396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83300f47e0 0x7f83301068d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f8320009830 tx=0x7f832000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:36.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.448+0000 7f8325ffb700 1 -- 172.21.15.152:0/3792394396 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8320009320 con 0x7f83300f47e0 2023-12-06T03:55:36.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.448+0000 7f8325ffb700 1 -- 172.21.15.152:0/3792394396 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83200136c0 con 0x7f83300f47e0 2023-12-06T03:55:36.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.448+0000 7f83377b9700 1 -- 172.21.15.152:0/3792394396 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8330106fc0 con 0x7f83300f47e0 2023-12-06T03:55:36.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.449+0000 7f8325ffb700 1 -- 172.21.15.152:0/3792394396 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f832001ee90 con 0x7f83300f47e0 2023-12-06T03:55:36.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.449+0000 7f83377b9700 1 -- 172.21.15.152:0/3792394396 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8330107450 con 0x7f83300f47e0 2023-12-06T03:55:36.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.449+0000 7f83377b9700 1 -- 172.21.15.152:0/3792394396 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8330045ad0 con 0x7f83300f47e0 2023-12-06T03:55:36.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.450+0000 7f8325ffb700 1 -- 172.21.15.152:0/3792394396 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f832001e420 con 0x7f83300f47e0 2023-12-06T03:55:36.453 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.450+0000 7f8325ffb700 1 --2- 172.21.15.152:0/3792394396 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f831c02ff10 0x7f831c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:36.453 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.450+0000 7f8325ffb700 1 -- 172.21.15.152:0/3792394396 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8320041080 con 0x7f83300f47e0 2023-12-06T03:55:36.453 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.450+0000 7f8334d54700 1 --2- 172.21.15.152:0/3792394396 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f831c02ff10 0x7f831c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:36.453 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.450+0000 7f8334d54700 1 --2- 172.21.15.152:0/3792394396 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f831c02ff10 0x7f831c0323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f832c0096a0 tx=0x7f832c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:36.456 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.454+0000 7f8325ffb700 1 -- 172.21.15.152:0/3792394396 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f832006d2b0 con 0x7f83300f47e0 2023-12-06T03:55:36.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.660+0000 7f83377b9700 1 -- 172.21.15.152:0/3792394396 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f833005eb50 con 0x7f83300f47e0 2023-12-06T03:55:36.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.661+0000 7f8325ffb700 1 -- 172.21.15.152:0/3792394396 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f832002c2d0 con 0x7f83300f47e0 2023-12-06T03:55:36.663 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:36.663 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:36.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.662+0000 7f83377b9700 1 -- 172.21.15.152:0/3792394396 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f831c02ff10 msgr2=0x7f831c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:36.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.662+0000 7f83377b9700 1 --2- 172.21.15.152:0/3792394396 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f831c02ff10 0x7f831c0323d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f832c0096a0 tx=0x7f832c006b40).stop 2023-12-06T03:55:36.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.662+0000 7f83377b9700 1 -- 172.21.15.152:0/3792394396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83300f47e0 msgr2=0x7f83301068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:36.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.662+0000 7f83377b9700 1 --2- 172.21.15.152:0/3792394396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83300f47e0 0x7f83301068d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f8320009830 tx=0x7f832000b040).stop 2023-12-06T03:55:36.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.662+0000 7f83377b9700 1 -- 172.21.15.152:0/3792394396 shutdown_connections 2023-12-06T03:55:36.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.662+0000 7f83377b9700 1 --2- 172.21.15.152:0/3792394396 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f831c02ff10 0x7f831c0323d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:36.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.662+0000 7f83377b9700 1 --2- 172.21.15.152:0/3792394396 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f83300f47e0 0x7f83301068d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:36.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.662+0000 7f83377b9700 1 -- 172.21.15.152:0/3792394396 >> 172.21.15.152:0/3792394396 conn(0x7f83300f0260 msgr2=0x7f83300f0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:36.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.663+0000 7f83377b9700 1 -- 172.21.15.152:0/3792394396 shutdown_connections 2023-12-06T03:55:36.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:36.663+0000 7f83377b9700 1 -- 172.21.15.152:0/3792394396 wait complete. 2023-12-06T03:55:36.665 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:37.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:36.407386+0000 mgr.smithi132.gokxja (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:37.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:36.662297+0000 mon.smithi132 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.152:0/3792394396' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:38.046 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:38.046 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:39.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.184+0000 7fdfaf59e700 1 -- 172.21.15.152:0/1184182556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb00f51a0 msgr2=0x7fdfb00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:39.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.184+0000 7fdfaf59e700 1 --2- 172.21.15.152:0/1184182556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb00f51a0 0x7fdfb00f55c0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fdfa4009fb0 tx=0x7fdfa4009b30).stop 2023-12-06T03:55:39.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.184+0000 7fdfaf59e700 1 -- 172.21.15.152:0/1184182556 shutdown_connections 2023-12-06T03:55:39.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.184+0000 7fdfaf59e700 1 --2- 172.21.15.152:0/1184182556 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb00f51a0 0x7fdfb00f55c0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:39.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.184+0000 7fdfaf59e700 1 -- 172.21.15.152:0/1184182556 >> 172.21.15.152:0/1184182556 conn(0x7fdfb00f09a0 msgr2=0x7fdfb00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:39.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.184+0000 7fdfaf59e700 1 -- 172.21.15.152:0/1184182556 shutdown_connections 2023-12-06T03:55:39.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.184+0000 7fdfaf59e700 1 -- 172.21.15.152:0/1184182556 wait complete. 2023-12-06T03:55:39.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.185+0000 7fdfaf59e700 1 Processor -- start 2023-12-06T03:55:39.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.185+0000 7fdfaf59e700 1 -- start start 2023-12-06T03:55:39.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.185+0000 7fdfaf59e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb00f51a0 0x7fdfb01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:39.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.185+0000 7fdfaf59e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfb01058a0 con 0x7fdfb00f51a0 2023-12-06T03:55:39.188 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.185+0000 7fdfae59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb00f51a0 0x7fdfb01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:39.189 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.185+0000 7fdfae59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb00f51a0 0x7fdfb01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:49816/0 (socket says 172.21.15.152:49816) 2023-12-06T03:55:39.189 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.185+0000 7fdfae59c700 1 -- 172.21.15.152:0/4265098742 learned_addr learned my addr 172.21.15.152:0/4265098742 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:39.189 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.186+0000 7fdfae59c700 1 -- 172.21.15.152:0/4265098742 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfa4005040 con 0x7fdfb00f51a0 2023-12-06T03:55:39.189 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.186+0000 7fdfae59c700 1 --2- 172.21.15.152:0/4265098742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb00f51a0 0x7fdfb01053b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fdfa4013510 tx=0x7fdfa400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:39.189 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.186+0000 7fdf9effd700 1 -- 172.21.15.152:0/4265098742 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa4009320 con 0x7fdfb00f51a0 2023-12-06T03:55:39.189 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.186+0000 7fdf9effd700 1 -- 172.21.15.152:0/4265098742 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfa4013f00 con 0x7fdfb00f51a0 2023-12-06T03:55:39.189 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.186+0000 7fdfaf59e700 1 -- 172.21.15.152:0/4265098742 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfb0105aa0 con 0x7fdfb00f51a0 2023-12-06T03:55:39.189 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.186+0000 7fdf9effd700 1 -- 172.21.15.152:0/4265098742 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfa4014dc0 con 0x7fdfb00f51a0 2023-12-06T03:55:39.189 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.186+0000 7fdfaf59e700 1 -- 172.21.15.152:0/4265098742 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfb0105f30 con 0x7fdfb00f51a0 2023-12-06T03:55:39.190 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.187+0000 7fdf9effd700 1 -- 172.21.15.152:0/4265098742 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfa40095f0 con 0x7fdfb00f51a0 2023-12-06T03:55:39.190 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.187+0000 7fdfaf59e700 1 -- 172.21.15.152:0/4265098742 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfb0045ad0 con 0x7fdfb00f51a0 2023-12-06T03:55:39.190 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.187+0000 7fdf9effd700 1 --2- 172.21.15.152:0/4265098742 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf9802ff60 0x7fdf98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:39.190 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.187+0000 7fdf9effd700 1 -- 172.21.15.152:0/4265098742 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfa40410c0 con 0x7fdfb00f51a0 2023-12-06T03:55:39.190 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.187+0000 7fdfadd9b700 1 --2- 172.21.15.152:0/4265098742 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf9802ff60 0x7fdf98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:39.190 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.187+0000 7fdfadd9b700 1 --2- 172.21.15.152:0/4265098742 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf9802ff60 0x7fdf98032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fdfa00096a0 tx=0x7fdfa0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:39.193 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.192+0000 7fdf9effd700 1 -- 172.21.15.152:0/4265098742 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfa401c790 con 0x7fdfb00f51a0 2023-12-06T03:55:39.401 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.399+0000 7fdfaf59e700 1 -- 172.21.15.152:0/4265098742 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfb0106de0 con 0x7fdfb00f51a0 2023-12-06T03:55:39.401 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.400+0000 7fdf9effd700 1 -- 172.21.15.152:0/4265098742 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdfa4010230 con 0x7fdfb00f51a0 2023-12-06T03:55:39.402 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:39.402 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:39.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.402+0000 7fdfaf59e700 1 -- 172.21.15.152:0/4265098742 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf9802ff60 msgr2=0x7fdf98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:39.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.402+0000 7fdfaf59e700 1 --2- 172.21.15.152:0/4265098742 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf9802ff60 0x7fdf98032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fdfa00096a0 tx=0x7fdfa0006b40).stop 2023-12-06T03:55:39.404 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.402+0000 7fdfaf59e700 1 -- 172.21.15.152:0/4265098742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb00f51a0 msgr2=0x7fdfb01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:39.404 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.402+0000 7fdfaf59e700 1 --2- 172.21.15.152:0/4265098742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb00f51a0 0x7fdfb01053b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fdfa4013510 tx=0x7fdfa400b040).stop 2023-12-06T03:55:39.404 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.402+0000 7fdfaf59e700 1 -- 172.21.15.152:0/4265098742 shutdown_connections 2023-12-06T03:55:39.404 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.402+0000 7fdfaf59e700 1 --2- 172.21.15.152:0/4265098742 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf9802ff60 0x7fdf98032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:39.404 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.402+0000 7fdfaf59e700 1 --2- 172.21.15.152:0/4265098742 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdfb00f51a0 0x7fdfb01053b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:39.404 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.402+0000 7fdfaf59e700 1 -- 172.21.15.152:0/4265098742 >> 172.21.15.152:0/4265098742 conn(0x7fdfb00f09a0 msgr2=0x7fdfb00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:39.404 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.402+0000 7fdfaf59e700 1 -- 172.21.15.152:0/4265098742 shutdown_connections 2023-12-06T03:55:39.404 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:39.402+0000 7fdfaf59e700 1 -- 172.21.15.152:0/4265098742 wait complete. 2023-12-06T03:55:39.404 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:39.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:39 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:38.407835+0000 mgr.smithi132.gokxja (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:39.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:39 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:39.401224+0000 mon.smithi132 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.152:0/4265098742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:40.782 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:40.783 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:41.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:41 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:40.408118+0000 mgr.smithi132.gokxja (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:41.900 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.897+0000 7f4956ba1700 1 -- 172.21.15.152:0/2862111114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49500f5410 msgr2=0x7f49500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:41.900 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.897+0000 7f4956ba1700 1 --2- 172.21.15.152:0/2862111114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49500f5410 0x7f49500f5830 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f4944009fb0 tx=0x7f4944009b30).stop 2023-12-06T03:55:41.900 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.898+0000 7f4956ba1700 1 -- 172.21.15.152:0/2862111114 shutdown_connections 2023-12-06T03:55:41.900 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.898+0000 7f4956ba1700 1 --2- 172.21.15.152:0/2862111114 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49500f5410 0x7f49500f5830 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:41.900 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.898+0000 7f4956ba1700 1 -- 172.21.15.152:0/2862111114 >> 172.21.15.152:0/2862111114 conn(0x7f49500f0b70 msgr2=0x7f49500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:41.900 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.898+0000 7f4956ba1700 1 -- 172.21.15.152:0/2862111114 shutdown_connections 2023-12-06T03:55:41.900 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.898+0000 7f4956ba1700 1 -- 172.21.15.152:0/2862111114 wait complete. 2023-12-06T03:55:41.900 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.898+0000 7f4956ba1700 1 Processor -- start 2023-12-06T03:55:41.900 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.898+0000 7f4956ba1700 1 -- start start 2023-12-06T03:55:41.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.898+0000 7f4956ba1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49500f5410 0x7f495010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:41.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.898+0000 7f4956ba1700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4950107240 con 0x7f49500f5410 2023-12-06T03:55:41.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.899+0000 7f495493d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49500f5410 0x7f495010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:41.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.899+0000 7f495493d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49500f5410 0x7f495010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:51350/0 (socket says 172.21.15.152:51350) 2023-12-06T03:55:41.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.899+0000 7f495493d700 1 -- 172.21.15.152:0/301285366 learned_addr learned my addr 172.21.15.152:0/301285366 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:41.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.899+0000 7f495493d700 1 -- 172.21.15.152:0/301285366 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4944005040 con 0x7f49500f5410 2023-12-06T03:55:41.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.899+0000 7f495493d700 1 --2- 172.21.15.152:0/301285366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49500f5410 0x7f495010b850 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f4944013510 tx=0x7f494400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:41.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.899+0000 7f494d7fa700 1 -- 172.21.15.152:0/301285366 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4944009320 con 0x7f49500f5410 2023-12-06T03:55:41.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.899+0000 7f4956ba1700 1 -- 172.21.15.152:0/301285366 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4950060220 con 0x7f49500f5410 2023-12-06T03:55:41.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.899+0000 7f494d7fa700 1 -- 172.21.15.152:0/301285366 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4944013f00 con 0x7f49500f5410 2023-12-06T03:55:41.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.899+0000 7f494d7fa700 1 -- 172.21.15.152:0/301285366 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4944014d90 con 0x7f49500f5410 2023-12-06T03:55:41.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.899+0000 7f4956ba1700 1 -- 172.21.15.152:0/301285366 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f495010bd40 con 0x7f49500f5410 2023-12-06T03:55:41.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.900+0000 7f494d7fa700 1 -- 172.21.15.152:0/301285366 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f49440095f0 con 0x7f49500f5410 2023-12-06T03:55:41.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.900+0000 7f4956ba1700 1 -- 172.21.15.152:0/301285366 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4950045ad0 con 0x7f49500f5410 2023-12-06T03:55:41.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.900+0000 7f494d7fa700 1 --2- 172.21.15.152:0/301285366 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f494002ff60 0x7f4940032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:41.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.900+0000 7f494d7fa700 1 -- 172.21.15.152:0/301285366 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4944041000 con 0x7f49500f5410 2023-12-06T03:55:41.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.901+0000 7f494ffff700 1 --2- 172.21.15.152:0/301285366 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f494002ff60 0x7f4940032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:41.904 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.901+0000 7f494ffff700 1 --2- 172.21.15.152:0/301285366 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f494002ff60 0x7f4940032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f493c0096a0 tx=0x7f493c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:41.907 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:41.905+0000 7f494d7fa700 1 -- 172.21.15.152:0/301285366 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f494401d7f0 con 0x7f49500f5410 2023-12-06T03:55:42.114 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:42.113+0000 7f4956ba1700 1 -- 172.21.15.152:0/301285366 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f49500fae60 con 0x7f49500f5410 2023-12-06T03:55:42.115 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:42.114+0000 7f494d7fa700 1 -- 172.21.15.152:0/301285366 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4944010230 con 0x7f49500f5410 2023-12-06T03:55:42.115 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:42.115 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:42.117 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:42.115+0000 7f4956ba1700 1 -- 172.21.15.152:0/301285366 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f494002ff60 msgr2=0x7f4940032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:42.117 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:42.115+0000 7f4956ba1700 1 --2- 172.21.15.152:0/301285366 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f494002ff60 0x7f4940032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f493c0096a0 tx=0x7f493c006b40).stop 2023-12-06T03:55:42.117 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:42.115+0000 7f4956ba1700 1 -- 172.21.15.152:0/301285366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49500f5410 msgr2=0x7f495010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:42.117 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:42.115+0000 7f4956ba1700 1 --2- 172.21.15.152:0/301285366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49500f5410 0x7f495010b850 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f4944013510 tx=0x7f494400b040).stop 2023-12-06T03:55:42.117 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:42.115+0000 7f4956ba1700 1 -- 172.21.15.152:0/301285366 shutdown_connections 2023-12-06T03:55:42.117 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:42.115+0000 7f4956ba1700 1 --2- 172.21.15.152:0/301285366 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f494002ff60 0x7f4940032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:42.117 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:42.115+0000 7f4956ba1700 1 --2- 172.21.15.152:0/301285366 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f49500f5410 0x7f495010b850 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:42.117 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:42.115+0000 7f4956ba1700 1 -- 172.21.15.152:0/301285366 >> 172.21.15.152:0/301285366 conn(0x7f49500f0b70 msgr2=0x7f49500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:42.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:42.115+0000 7f4956ba1700 1 -- 172.21.15.152:0/301285366 shutdown_connections 2023-12-06T03:55:42.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:42.115+0000 7f4956ba1700 1 -- 172.21.15.152:0/301285366 wait complete. 2023-12-06T03:55:42.118 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:42.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:42 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:42.114845+0000 mon.smithi132 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.152:0/301285366' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:43.483 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:43.484 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:43.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:42.408473+0000 mgr.smithi132.gokxja (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:44.591 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.588+0000 7fd31f1d7700 1 -- 172.21.15.152:0/1006369363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3180f5410 msgr2=0x7fd3180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:44.591 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.588+0000 7fd31f1d7700 1 --2- 172.21.15.152:0/1006369363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3180f5410 0x7fd3180f5830 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7fd30c009fb0 tx=0x7fd30c009b30).stop 2023-12-06T03:55:44.591 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.588+0000 7fd31f1d7700 1 -- 172.21.15.152:0/1006369363 shutdown_connections 2023-12-06T03:55:44.591 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.588+0000 7fd31f1d7700 1 --2- 172.21.15.152:0/1006369363 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3180f5410 0x7fd3180f5830 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:44.591 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.588+0000 7fd31f1d7700 1 -- 172.21.15.152:0/1006369363 >> 172.21.15.152:0/1006369363 conn(0x7fd3180f0b70 msgr2=0x7fd3180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:44.591 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.588+0000 7fd31f1d7700 1 -- 172.21.15.152:0/1006369363 shutdown_connections 2023-12-06T03:55:44.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.588+0000 7fd31f1d7700 1 -- 172.21.15.152:0/1006369363 wait complete. 2023-12-06T03:55:44.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.589+0000 7fd31f1d7700 1 Processor -- start 2023-12-06T03:55:44.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.589+0000 7fd31f1d7700 1 -- start start 2023-12-06T03:55:44.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.589+0000 7fd31f1d7700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3180f5410 0x7fd318109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:44.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.589+0000 7fd31f1d7700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd318060220 con 0x7fd3180f5410 2023-12-06T03:55:44.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.589+0000 7fd31cf73700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3180f5410 0x7fd318109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:44.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.589+0000 7fd31cf73700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3180f5410 0x7fd318109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:51374/0 (socket says 172.21.15.152:51374) 2023-12-06T03:55:44.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.589+0000 7fd31cf73700 1 -- 172.21.15.152:0/2621588189 learned_addr learned my addr 172.21.15.152:0/2621588189 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:44.593 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.590+0000 7fd31cf73700 1 -- 172.21.15.152:0/2621588189 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30c005040 con 0x7fd3180f5410 2023-12-06T03:55:44.593 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.590+0000 7fd31cf73700 1 --2- 172.21.15.152:0/2621588189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3180f5410 0x7fd318109420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fd30c013510 tx=0x7fd30c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:44.593 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.590+0000 7fd3157fa700 1 -- 172.21.15.152:0/2621588189 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd30c009320 con 0x7fd3180f5410 2023-12-06T03:55:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.590+0000 7fd31f1d7700 1 -- 172.21.15.152:0/2621588189 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd318060420 con 0x7fd3180f5410 2023-12-06T03:55:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.590+0000 7fd3157fa700 1 -- 172.21.15.152:0/2621588189 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd30c013f00 con 0x7fd3180f5410 2023-12-06T03:55:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.590+0000 7fd3157fa700 1 -- 172.21.15.152:0/2621588189 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd30c014d90 con 0x7fd3180f5410 2023-12-06T03:55:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.590+0000 7fd31f1d7700 1 -- 172.21.15.152:0/2621588189 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd318109ad0 con 0x7fd3180f5410 2023-12-06T03:55:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.591+0000 7fd3157fa700 1 -- 172.21.15.152:0/2621588189 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd30c0095f0 con 0x7fd3180f5410 2023-12-06T03:55:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.591+0000 7fd31f1d7700 1 -- 172.21.15.152:0/2621588189 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd318059100 con 0x7fd3180f5410 2023-12-06T03:55:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.591+0000 7fd3157fa700 1 --2- 172.21.15.152:0/2621588189 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd30802ff60 0x7fd308032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.591+0000 7fd3157fa700 1 -- 172.21.15.152:0/2621588189 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd30c040fe0 con 0x7fd3180f5410 2023-12-06T03:55:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.591+0000 7fd317fff700 1 --2- 172.21.15.152:0/2621588189 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd30802ff60 0x7fd308032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:44.594 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.592+0000 7fd317fff700 1 --2- 172.21.15.152:0/2621588189 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd30802ff60 0x7fd308032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fd3040096a0 tx=0x7fd304006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:44.597 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.596+0000 7fd3157fa700 1 -- 172.21.15.152:0/2621588189 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd30c010230 con 0x7fd3180f5410 2023-12-06T03:55:44.805 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.803+0000 7fd31f1d7700 1 -- 172.21.15.152:0/2621588189 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd318045ad0 con 0x7fd3180f5410 2023-12-06T03:55:44.806 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.804+0000 7fd3157fa700 1 -- 172.21.15.152:0/2621588189 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd30c010230 con 0x7fd3180f5410 2023-12-06T03:55:44.806 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:44.806 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:44.807 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.806+0000 7fd31f1d7700 1 -- 172.21.15.152:0/2621588189 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd30802ff60 msgr2=0x7fd308032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:44.807 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.806+0000 7fd31f1d7700 1 --2- 172.21.15.152:0/2621588189 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd30802ff60 0x7fd308032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fd3040096a0 tx=0x7fd304006b40).stop 2023-12-06T03:55:44.807 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.806+0000 7fd31f1d7700 1 -- 172.21.15.152:0/2621588189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3180f5410 msgr2=0x7fd318109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:44.808 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.806+0000 7fd31f1d7700 1 --2- 172.21.15.152:0/2621588189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3180f5410 0x7fd318109420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fd30c013510 tx=0x7fd30c00b040).stop 2023-12-06T03:55:44.808 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.806+0000 7fd31f1d7700 1 -- 172.21.15.152:0/2621588189 shutdown_connections 2023-12-06T03:55:44.808 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.806+0000 7fd31f1d7700 1 --2- 172.21.15.152:0/2621588189 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd30802ff60 0x7fd308032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:44.808 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.806+0000 7fd31f1d7700 1 --2- 172.21.15.152:0/2621588189 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd3180f5410 0x7fd318109420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:44.808 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.806+0000 7fd31f1d7700 1 -- 172.21.15.152:0/2621588189 >> 172.21.15.152:0/2621588189 conn(0x7fd3180f0b70 msgr2=0x7fd3180f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:44.808 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.806+0000 7fd31f1d7700 1 -- 172.21.15.152:0/2621588189 shutdown_connections 2023-12-06T03:55:44.808 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:44.806+0000 7fd31f1d7700 1 -- 172.21.15.152:0/2621588189 wait complete. 2023-12-06T03:55:44.808 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:45.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:45 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:44.408724+0000 mgr.smithi132.gokxja (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:45.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:45 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:44.805426+0000 mon.smithi132 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.152:0/2621588189' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:46.177 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:46.177 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:47.289 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.287+0000 7fb09f268700 1 -- 172.21.15.152:0/3689240029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0980f6510 msgr2=0x7fb0980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:47.289 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.287+0000 7fb09f268700 1 --2- 172.21.15.152:0/3689240029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0980f6510 0x7fb0980f6930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fb094009fb0 tx=0x7fb094009b30).stop 2023-12-06T03:55:47.289 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.287+0000 7fb09f268700 1 -- 172.21.15.152:0/3689240029 shutdown_connections 2023-12-06T03:55:47.289 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.287+0000 7fb09f268700 1 --2- 172.21.15.152:0/3689240029 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0980f6510 0x7fb0980f6930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:47.289 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.287+0000 7fb09f268700 1 -- 172.21.15.152:0/3689240029 >> 172.21.15.152:0/3689240029 conn(0x7fb0980f1c70 msgr2=0x7fb0980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:47.289 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.287+0000 7fb09f268700 1 -- 172.21.15.152:0/3689240029 shutdown_connections 2023-12-06T03:55:47.290 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.287+0000 7fb09f268700 1 -- 172.21.15.152:0/3689240029 wait complete. 2023-12-06T03:55:47.291 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.288+0000 7fb09f268700 1 Processor -- start 2023-12-06T03:55:47.291 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.288+0000 7fb09f268700 1 -- start start 2023-12-06T03:55:47.291 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.288+0000 7fb09f268700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0980f6510 0x7fb098106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:47.291 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.288+0000 7fb09f268700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb098106c20 con 0x7fb0980f6510 2023-12-06T03:55:47.291 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.288+0000 7fb09d004700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0980f6510 0x7fb098106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:47.291 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.288+0000 7fb09d004700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0980f6510 0x7fb098106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:51398/0 (socket says 172.21.15.152:51398) 2023-12-06T03:55:47.291 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.288+0000 7fb09d004700 1 -- 172.21.15.152:0/3222391043 learned_addr learned my addr 172.21.15.152:0/3222391043 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:47.291 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.289+0000 7fb09d004700 1 -- 172.21.15.152:0/3222391043 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb094005040 con 0x7fb0980f6510 2023-12-06T03:55:47.291 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.289+0000 7fb09d004700 1 --2- 172.21.15.152:0/3222391043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0980f6510 0x7fb098106730 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb094013510 tx=0x7fb09400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:47.291 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.289+0000 7fb089ffb700 1 -- 172.21.15.152:0/3222391043 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb094009320 con 0x7fb0980f6510 2023-12-06T03:55:47.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.289+0000 7fb09f268700 1 -- 172.21.15.152:0/3222391043 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb098106e20 con 0x7fb0980f6510 2023-12-06T03:55:47.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.289+0000 7fb089ffb700 1 -- 172.21.15.152:0/3222391043 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb094013f00 con 0x7fb0980f6510 2023-12-06T03:55:47.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.289+0000 7fb089ffb700 1 -- 172.21.15.152:0/3222391043 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb094014d90 con 0x7fb0980f6510 2023-12-06T03:55:47.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.289+0000 7fb09f268700 1 -- 172.21.15.152:0/3222391043 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0981072b0 con 0x7fb0980f6510 2023-12-06T03:55:47.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.290+0000 7fb089ffb700 1 -- 172.21.15.152:0/3222391043 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0940263a0 con 0x7fb0980f6510 2023-12-06T03:55:47.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.290+0000 7fb09f268700 1 -- 172.21.15.152:0/3222391043 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb098045ad0 con 0x7fb0980f6510 2023-12-06T03:55:47.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.290+0000 7fb089ffb700 1 --2- 172.21.15.152:0/3222391043 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb08402ffb0 0x7fb084032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:47.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.290+0000 7fb089ffb700 1 -- 172.21.15.152:0/3222391043 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb0940410c0 con 0x7fb0980f6510 2023-12-06T03:55:47.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.291+0000 7fb09c803700 1 --2- 172.21.15.152:0/3222391043 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb08402ffb0 0x7fb084032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:47.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.291+0000 7fb09c803700 1 --2- 172.21.15.152:0/3222391043 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb08402ffb0 0x7fb084032470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fb08c0096a0 tx=0x7fb08c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:47.297 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.295+0000 7fb089ffb700 1 -- 172.21.15.152:0/3222391043 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0940095f0 con 0x7fb0980f6510 2023-12-06T03:55:47.505 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.503+0000 7fb09f268700 1 -- 172.21.15.152:0/3222391043 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0980fbf60 con 0x7fb0980f6510 2023-12-06T03:55:47.505 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.504+0000 7fb089ffb700 1 -- 172.21.15.152:0/3222391043 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb09401db70 con 0x7fb0980f6510 2023-12-06T03:55:47.505 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:47.506 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:47.507 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.505+0000 7fb09f268700 1 -- 172.21.15.152:0/3222391043 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb08402ffb0 msgr2=0x7fb084032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:47.507 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.505+0000 7fb09f268700 1 --2- 172.21.15.152:0/3222391043 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb08402ffb0 0x7fb084032470 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fb08c0096a0 tx=0x7fb08c006b40).stop 2023-12-06T03:55:47.507 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.505+0000 7fb09f268700 1 -- 172.21.15.152:0/3222391043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0980f6510 msgr2=0x7fb098106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:47.507 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.505+0000 7fb09f268700 1 --2- 172.21.15.152:0/3222391043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0980f6510 0x7fb098106730 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fb094013510 tx=0x7fb09400b040).stop 2023-12-06T03:55:47.507 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.505+0000 7fb09f268700 1 -- 172.21.15.152:0/3222391043 shutdown_connections 2023-12-06T03:55:47.507 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.505+0000 7fb09f268700 1 --2- 172.21.15.152:0/3222391043 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb08402ffb0 0x7fb084032470 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:47.508 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.505+0000 7fb09f268700 1 --2- 172.21.15.152:0/3222391043 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0980f6510 0x7fb098106730 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:47.508 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.505+0000 7fb09f268700 1 -- 172.21.15.152:0/3222391043 >> 172.21.15.152:0/3222391043 conn(0x7fb0980f1c70 msgr2=0x7fb0980f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:47.508 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.505+0000 7fb09f268700 1 -- 172.21.15.152:0/3222391043 shutdown_connections 2023-12-06T03:55:47.508 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:47.505+0000 7fb09f268700 1 -- 172.21.15.152:0/3222391043 wait complete. 2023-12-06T03:55:47.508 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:47.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:47 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:46.409024+0000 mgr.smithi132.gokxja (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:47.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:47 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:47.505039+0000 mon.smithi132 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.152:0/3222391043' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:48.909 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:48.910 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:49.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:49 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:48.409288+0000 mgr.smithi132.gokxja (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:50.010 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.008+0000 7fb2bde9b700 1 -- 172.21.15.152:0/715891061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2b80f6510 msgr2=0x7fb2b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:50.011 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.008+0000 7fb2bde9b700 1 --2- 172.21.15.152:0/715891061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2b80f6510 0x7fb2b80f6930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fb2ac009fb0 tx=0x7fb2ac009b30).stop 2023-12-06T03:55:50.011 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.008+0000 7fb2bde9b700 1 -- 172.21.15.152:0/715891061 shutdown_connections 2023-12-06T03:55:50.011 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.008+0000 7fb2bde9b700 1 --2- 172.21.15.152:0/715891061 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2b80f6510 0x7fb2b80f6930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:50.011 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.008+0000 7fb2bde9b700 1 -- 172.21.15.152:0/715891061 >> 172.21.15.152:0/715891061 conn(0x7fb2b80f1c70 msgr2=0x7fb2b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:50.011 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.008+0000 7fb2bde9b700 1 -- 172.21.15.152:0/715891061 shutdown_connections 2023-12-06T03:55:50.011 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.008+0000 7fb2bde9b700 1 -- 172.21.15.152:0/715891061 wait complete. 2023-12-06T03:55:50.011 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.009+0000 7fb2bde9b700 1 Processor -- start 2023-12-06T03:55:50.011 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.009+0000 7fb2bde9b700 1 -- start start 2023-12-06T03:55:50.011 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.009+0000 7fb2bde9b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2b80f6510 0x7fb2b8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:50.011 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.009+0000 7fb2bde9b700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2b8106c20 con 0x7fb2b80f6510 2023-12-06T03:55:50.013 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.009+0000 7fb2b77fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2b80f6510 0x7fb2b8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:50.013 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.009+0000 7fb2b77fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2b80f6510 0x7fb2b8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:44344/0 (socket says 172.21.15.152:44344) 2023-12-06T03:55:50.013 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.009+0000 7fb2b77fe700 1 -- 172.21.15.152:0/1096198132 learned_addr learned my addr 172.21.15.152:0/1096198132 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:50.013 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.010+0000 7fb2b77fe700 1 -- 172.21.15.152:0/1096198132 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2ac005040 con 0x7fb2b80f6510 2023-12-06T03:55:50.013 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.010+0000 7fb2b77fe700 1 --2- 172.21.15.152:0/1096198132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2b80f6510 0x7fb2b8106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fb2ac014000 tx=0x7fb2ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:50.013 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.010+0000 7fb29ffff700 1 -- 172.21.15.152:0/1096198132 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2ac0095f0 con 0x7fb2b80f6510 2023-12-06T03:55:50.013 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.010+0000 7fb2bde9b700 1 -- 172.21.15.152:0/1096198132 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2b8106e20 con 0x7fb2b80f6510 2023-12-06T03:55:50.013 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.010+0000 7fb29ffff700 1 -- 172.21.15.152:0/1096198132 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2ac007e70 con 0x7fb2b80f6510 2023-12-06T03:55:50.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.010+0000 7fb29ffff700 1 -- 172.21.15.152:0/1096198132 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2ac014ea0 con 0x7fb2b80f6510 2023-12-06T03:55:50.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.010+0000 7fb2bde9b700 1 -- 172.21.15.152:0/1096198132 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2b81072b0 con 0x7fb2b80f6510 2023-12-06T03:55:50.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.011+0000 7fb29ffff700 1 -- 172.21.15.152:0/1096198132 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2ac0079c0 con 0x7fb2b80f6510 2023-12-06T03:55:50.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.011+0000 7fb2bde9b700 1 -- 172.21.15.152:0/1096198132 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2b8045ad0 con 0x7fb2b80f6510 2023-12-06T03:55:50.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.011+0000 7fb29ffff700 1 --2- 172.21.15.152:0/1096198132 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb2a002ff60 0x7fb2a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:50.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.011+0000 7fb29ffff700 1 -- 172.21.15.152:0/1096198132 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb2ac040920 con 0x7fb2b80f6510 2023-12-06T03:55:50.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.011+0000 7fb2b6ffd700 1 --2- 172.21.15.152:0/1096198132 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb2a002ff60 0x7fb2a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:50.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.012+0000 7fb2b6ffd700 1 --2- 172.21.15.152:0/1096198132 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb2a002ff60 0x7fb2a0032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fb2a80096a0 tx=0x7fb2a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:50.017 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.016+0000 7fb29ffff700 1 -- 172.21.15.152:0/1096198132 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2ac00f950 con 0x7fb2b80f6510 2023-12-06T03:55:50.222 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.219+0000 7fb2bde9b700 1 -- 172.21.15.152:0/1096198132 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2b80fbf60 con 0x7fb2b80f6510 2023-12-06T03:55:50.222 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.220+0000 7fb29ffff700 1 -- 172.21.15.152:0/1096198132 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb2ac010230 con 0x7fb2b80f6510 2023-12-06T03:55:50.222 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:50.222 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:50.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.222+0000 7fb2bde9b700 1 -- 172.21.15.152:0/1096198132 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb2a002ff60 msgr2=0x7fb2a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:50.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.222+0000 7fb2bde9b700 1 --2- 172.21.15.152:0/1096198132 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb2a002ff60 0x7fb2a0032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fb2a80096a0 tx=0x7fb2a8006b40).stop 2023-12-06T03:55:50.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.222+0000 7fb2bde9b700 1 -- 172.21.15.152:0/1096198132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2b80f6510 msgr2=0x7fb2b8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:50.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.222+0000 7fb2bde9b700 1 --2- 172.21.15.152:0/1096198132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2b80f6510 0x7fb2b8106730 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fb2ac014000 tx=0x7fb2ac00b040).stop 2023-12-06T03:55:50.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.222+0000 7fb2bde9b700 1 -- 172.21.15.152:0/1096198132 shutdown_connections 2023-12-06T03:55:50.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.222+0000 7fb2bde9b700 1 --2- 172.21.15.152:0/1096198132 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb2a002ff60 0x7fb2a0032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:50.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.222+0000 7fb2bde9b700 1 --2- 172.21.15.152:0/1096198132 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2b80f6510 0x7fb2b8106730 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:50.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.222+0000 7fb2bde9b700 1 -- 172.21.15.152:0/1096198132 >> 172.21.15.152:0/1096198132 conn(0x7fb2b80f1c70 msgr2=0x7fb2b80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:50.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.222+0000 7fb2bde9b700 1 -- 172.21.15.152:0/1096198132 shutdown_connections 2023-12-06T03:55:50.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:50.222+0000 7fb2bde9b700 1 -- 172.21.15.152:0/1096198132 wait complete. 2023-12-06T03:55:50.224 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:50.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:50 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:50.221584+0000 mon.smithi132 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.152:0/1096198132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:51.600 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:51.600 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:51.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:51 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:50.409645+0000 mgr.smithi132.gokxja (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:52.715 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.713+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/1339713545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc80f5410 msgr2=0x7f8dc80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:52.715 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.713+0000 7f8dcd5b9700 1 --2- 172.21.15.152:0/1339713545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc80f5410 0x7f8dc80f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f8dbc009fb0 tx=0x7f8dbc009b30).stop 2023-12-06T03:55:52.715 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.713+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/1339713545 shutdown_connections 2023-12-06T03:55:52.715 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.713+0000 7f8dcd5b9700 1 --2- 172.21.15.152:0/1339713545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc80f5410 0x7f8dc80f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:52.715 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.713+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/1339713545 >> 172.21.15.152:0/1339713545 conn(0x7f8dc80f0b70 msgr2=0x7f8dc80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:52.716 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.713+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/1339713545 shutdown_connections 2023-12-06T03:55:52.716 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.713+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/1339713545 wait complete. 2023-12-06T03:55:52.716 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.714+0000 7f8dcd5b9700 1 Processor -- start 2023-12-06T03:55:52.716 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.714+0000 7f8dcd5b9700 1 -- start start 2023-12-06T03:55:52.717 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.714+0000 7f8dcd5b9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc80f5410 0x7f8dc8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:52.717 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.714+0000 7f8dcd5b9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc8105b20 con 0x7f8dc80f5410 2023-12-06T03:55:52.717 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.714+0000 7f8dc6ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc80f5410 0x7f8dc8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:52.717 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.714+0000 7f8dc6ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc80f5410 0x7f8dc8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:44370/0 (socket says 172.21.15.152:44370) 2023-12-06T03:55:52.717 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.714+0000 7f8dc6ffd700 1 -- 172.21.15.152:0/660489752 learned_addr learned my addr 172.21.15.152:0/660489752 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:52.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.714+0000 7f8dc6ffd700 1 -- 172.21.15.152:0/660489752 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8dbc005040 con 0x7f8dc80f5410 2023-12-06T03:55:52.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.714+0000 7f8dc6ffd700 1 --2- 172.21.15.152:0/660489752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc80f5410 0x7f8dc8105630 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f8dbc014000 tx=0x7f8dbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:52.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.715+0000 7f8daf7fe700 1 -- 172.21.15.152:0/660489752 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dbc0095f0 con 0x7f8dc80f5410 2023-12-06T03:55:52.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.715+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/660489752 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8dc8105d20 con 0x7f8dc80f5410 2023-12-06T03:55:52.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.715+0000 7f8daf7fe700 1 -- 172.21.15.152:0/660489752 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8dbc007e70 con 0x7f8dc80f5410 2023-12-06T03:55:52.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.715+0000 7f8daf7fe700 1 -- 172.21.15.152:0/660489752 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8dbc014ea0 con 0x7f8dc80f5410 2023-12-06T03:55:52.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.715+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/660489752 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8dc81061b0 con 0x7f8dc80f5410 2023-12-06T03:55:52.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.715+0000 7f8daf7fe700 1 -- 172.21.15.152:0/660489752 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8dbc027070 con 0x7f8dc80f5410 2023-12-06T03:55:52.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.716+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/660489752 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8dc8045ad0 con 0x7f8dc80f5410 2023-12-06T03:55:52.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.716+0000 7f8daf7fe700 1 --2- 172.21.15.152:0/660489752 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8db002ffb0 0x7f8db0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:52.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.716+0000 7f8daf7fe700 1 -- 172.21.15.152:0/660489752 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8dbc042790 con 0x7f8dc80f5410 2023-12-06T03:55:52.719 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.716+0000 7f8dc67fc700 1 --2- 172.21.15.152:0/660489752 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8db002ffb0 0x7f8db0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:52.719 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.716+0000 7f8dc67fc700 1 --2- 172.21.15.152:0/660489752 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8db002ffb0 0x7f8db0032470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f8db80096a0 tx=0x7f8db8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:52.722 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.720+0000 7f8daf7fe700 1 -- 172.21.15.152:0/660489752 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8dbc0274e0 con 0x7f8dc80f5410 2023-12-06T03:55:52.929 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.927+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/660489752 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8dc80fae60 con 0x7f8dc80f5410 2023-12-06T03:55:52.930 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.928+0000 7f8daf7fe700 1 -- 172.21.15.152:0/660489752 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8dbc027300 con 0x7f8dc80f5410 2023-12-06T03:55:52.930 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:52.930 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:52.931 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.930+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/660489752 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8db002ffb0 msgr2=0x7f8db0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:52.932 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.930+0000 7f8dcd5b9700 1 --2- 172.21.15.152:0/660489752 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8db002ffb0 0x7f8db0032470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f8db80096a0 tx=0x7f8db8006b40).stop 2023-12-06T03:55:52.932 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.930+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/660489752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc80f5410 msgr2=0x7f8dc8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:52.932 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.930+0000 7f8dcd5b9700 1 --2- 172.21.15.152:0/660489752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc80f5410 0x7f8dc8105630 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f8dbc014000 tx=0x7f8dbc00b040).stop 2023-12-06T03:55:52.932 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.930+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/660489752 shutdown_connections 2023-12-06T03:55:52.932 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.930+0000 7f8dcd5b9700 1 --2- 172.21.15.152:0/660489752 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8db002ffb0 0x7f8db0032470 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:52.932 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.930+0000 7f8dcd5b9700 1 --2- 172.21.15.152:0/660489752 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8dc80f5410 0x7f8dc8105630 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:52.932 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.930+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/660489752 >> 172.21.15.152:0/660489752 conn(0x7f8dc80f0b70 msgr2=0x7f8dc80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:52.932 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.930+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/660489752 shutdown_connections 2023-12-06T03:55:52.932 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:52.930+0000 7f8dcd5b9700 1 -- 172.21.15.152:0/660489752 wait complete. 2023-12-06T03:55:52.932 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:53.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:53 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:52.409980+0000 mgr.smithi132.gokxja (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:53.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:53 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:52.929376+0000 mon.smithi132 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.152:0/660489752' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:54.305 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:54.305 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:54.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:54.518290+0000 mon.smithi132 (mon.0) 176 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/mirror_snapshot_schedule"}]: dispatch 2023-12-06T03:55:54.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:54.526977+0000 mon.smithi132 (mon.0) 177 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/trash_purge_schedule"}]: dispatch 2023-12-06T03:55:55.447 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.446+0000 7feb9950e700 1 -- 172.21.15.152:0/2918275153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f5410 msgr2=0x7feb940f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:55.448 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.446+0000 7feb9950e700 1 --2- 172.21.15.152:0/2918275153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f5410 0x7feb940f5830 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7feb88009fb0 tx=0x7feb88009b30).stop 2023-12-06T03:55:55.448 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.446+0000 7feb9950e700 1 -- 172.21.15.152:0/2918275153 shutdown_connections 2023-12-06T03:55:55.448 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.446+0000 7feb9950e700 1 --2- 172.21.15.152:0/2918275153 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f5410 0x7feb940f5830 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:55.448 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.446+0000 7feb9950e700 1 -- 172.21.15.152:0/2918275153 >> 172.21.15.152:0/2918275153 conn(0x7feb940f0b70 msgr2=0x7feb940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:55.448 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.446+0000 7feb9950e700 1 -- 172.21.15.152:0/2918275153 shutdown_connections 2023-12-06T03:55:55.448 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.446+0000 7feb9950e700 1 -- 172.21.15.152:0/2918275153 wait complete. 2023-12-06T03:55:55.449 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.446+0000 7feb9950e700 1 Processor -- start 2023-12-06T03:55:55.449 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.446+0000 7feb9950e700 1 -- start start 2023-12-06T03:55:55.449 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.447+0000 7feb9950e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f5410 0x7feb94106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:55.449 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.447+0000 7feb9950e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb94106c00 con 0x7feb940f5410 2023-12-06T03:55:55.449 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.447+0000 7feb92ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f5410 0x7feb94106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:55.449 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.447+0000 7feb92ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f5410 0x7feb94106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:44394/0 (socket says 172.21.15.152:44394) 2023-12-06T03:55:55.449 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.447+0000 7feb92ffd700 1 -- 172.21.15.152:0/1509021637 learned_addr learned my addr 172.21.15.152:0/1509021637 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:55.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.447+0000 7feb92ffd700 1 -- 172.21.15.152:0/1509021637 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb88005040 con 0x7feb940f5410 2023-12-06T03:55:55.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.447+0000 7feb92ffd700 1 --2- 172.21.15.152:0/1509021637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f5410 0x7feb94106710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7feb88013510 tx=0x7feb8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:55.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.447+0000 7feb7b7fe700 1 -- 172.21.15.152:0/1509021637 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb88009320 con 0x7feb940f5410 2023-12-06T03:55:55.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.447+0000 7feb9950e700 1 -- 172.21.15.152:0/1509021637 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb94106e00 con 0x7feb940f5410 2023-12-06T03:55:55.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.448+0000 7feb7b7fe700 1 -- 172.21.15.152:0/1509021637 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb88013f00 con 0x7feb940f5410 2023-12-06T03:55:55.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.448+0000 7feb7b7fe700 1 -- 172.21.15.152:0/1509021637 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb88014d90 con 0x7feb940f5410 2023-12-06T03:55:55.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.448+0000 7feb9950e700 1 -- 172.21.15.152:0/1509021637 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb94107290 con 0x7feb940f5410 2023-12-06T03:55:55.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.448+0000 7feb7b7fe700 1 -- 172.21.15.152:0/1509021637 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb880263a0 con 0x7feb940f5410 2023-12-06T03:55:55.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.448+0000 7feb7b7fe700 1 --2- 172.21.15.152:0/1509021637 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb7c02ff60 0x7feb7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:55.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.449+0000 7feb7b7fe700 1 -- 172.21.15.152:0/1509021637 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb880413b0 con 0x7feb940f5410 2023-12-06T03:55:55.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.449+0000 7feb9950e700 1 -- 172.21.15.152:0/1509021637 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb94045ad0 con 0x7feb940f5410 2023-12-06T03:55:55.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.449+0000 7feb927fc700 1 --2- 172.21.15.152:0/1509021637 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb7c02ff60 0x7feb7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:55.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.449+0000 7feb927fc700 1 --2- 172.21.15.152:0/1509021637 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb7c02ff60 0x7feb7c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7feb840096a0 tx=0x7feb84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:55.455 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.453+0000 7feb7b7fe700 1 -- 172.21.15.152:0/1509021637 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb88012920 con 0x7feb940f5410 2023-12-06T03:55:55.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.662+0000 7feb9950e700 1 -- 172.21.15.152:0/1509021637 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb940fae60 con 0x7feb940f5410 2023-12-06T03:55:55.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.662+0000 7feb7b7fe700 1 -- 172.21.15.152:0/1509021637 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feb8802b030 con 0x7feb940f5410 2023-12-06T03:55:55.665 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:55.665 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:55.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.664+0000 7feb9950e700 1 -- 172.21.15.152:0/1509021637 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb7c02ff60 msgr2=0x7feb7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:55.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.664+0000 7feb9950e700 1 --2- 172.21.15.152:0/1509021637 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb7c02ff60 0x7feb7c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7feb840096a0 tx=0x7feb84006b40).stop 2023-12-06T03:55:55.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.664+0000 7feb9950e700 1 -- 172.21.15.152:0/1509021637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f5410 msgr2=0x7feb94106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:55.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.664+0000 7feb9950e700 1 --2- 172.21.15.152:0/1509021637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f5410 0x7feb94106710 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7feb88013510 tx=0x7feb8800b040).stop 2023-12-06T03:55:55.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.664+0000 7feb9950e700 1 -- 172.21.15.152:0/1509021637 shutdown_connections 2023-12-06T03:55:55.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.664+0000 7feb9950e700 1 --2- 172.21.15.152:0/1509021637 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb7c02ff60 0x7feb7c032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:55.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.664+0000 7feb9950e700 1 --2- 172.21.15.152:0/1509021637 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb940f5410 0x7feb94106710 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:55.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.664+0000 7feb9950e700 1 -- 172.21.15.152:0/1509021637 >> 172.21.15.152:0/1509021637 conn(0x7feb940f0b70 msgr2=0x7feb9405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:55.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.664+0000 7feb9950e700 1 -- 172.21.15.152:0/1509021637 shutdown_connections 2023-12-06T03:55:55.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:55.664+0000 7feb9950e700 1 -- 172.21.15.152:0/1509021637 wait complete. 2023-12-06T03:55:55.667 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:55.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:54.410306+0000 mgr.smithi132.gokxja (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:56.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:56 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:55.663877+0000 mon.smithi132 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.152:0/1509021637' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:57.053 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:57.054 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:57.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:56.410581+0000 mgr.smithi132.gokxja (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:55:58.139 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.136+0000 7f4e185b9700 1 -- 172.21.15.152:0/2186266813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e10060280 msgr2=0x7f4e100f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:58.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.136+0000 7f4e185b9700 1 --2- 172.21.15.152:0/2186266813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e10060280 0x7f4e100f90f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f4e0c009fb0 tx=0x7f4e0c009b30).stop 2023-12-06T03:55:58.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.136+0000 7f4e185b9700 1 -- 172.21.15.152:0/2186266813 shutdown_connections 2023-12-06T03:55:58.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.136+0000 7f4e185b9700 1 --2- 172.21.15.152:0/2186266813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e10060280 0x7f4e100f90f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:58.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.136+0000 7f4e185b9700 1 -- 172.21.15.152:0/2186266813 >> 172.21.15.152:0/2186266813 conn(0x7f4e100f0b70 msgr2=0x7f4e100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:58.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.136+0000 7f4e185b9700 1 -- 172.21.15.152:0/2186266813 shutdown_connections 2023-12-06T03:55:58.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.136+0000 7f4e185b9700 1 -- 172.21.15.152:0/2186266813 wait complete. 2023-12-06T03:55:58.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.136+0000 7f4e185b9700 1 Processor -- start 2023-12-06T03:55:58.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.136+0000 7f4e185b9700 1 -- start start 2023-12-06T03:55:58.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.137+0000 7f4e185b9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e10060280 0x7f4e101066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:58.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.137+0000 7f4e185b9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e10106bd0 con 0x7f4e10060280 2023-12-06T03:55:58.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.137+0000 7f4e16355700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e10060280 0x7f4e101066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:58.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.137+0000 7f4e16355700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e10060280 0x7f4e101066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:44414/0 (socket says 172.21.15.152:44414) 2023-12-06T03:55:58.141 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.137+0000 7f4e16355700 1 -- 172.21.15.152:0/2855111546 learned_addr learned my addr 172.21.15.152:0/2855111546 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:55:58.141 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.137+0000 7f4e16355700 1 -- 172.21.15.152:0/2855111546 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c005040 con 0x7f4e10060280 2023-12-06T03:55:58.141 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.137+0000 7f4e16355700 1 --2- 172.21.15.152:0/2855111546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e10060280 0x7f4e101066e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f4e0c013510 tx=0x7f4e0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:58.141 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.137+0000 7f4e06ffd700 1 -- 172.21.15.152:0/2855111546 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e0c009320 con 0x7f4e10060280 2023-12-06T03:55:58.141 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.137+0000 7f4e185b9700 1 -- 172.21.15.152:0/2855111546 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e10106dd0 con 0x7f4e10060280 2023-12-06T03:55:58.141 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.137+0000 7f4e06ffd700 1 -- 172.21.15.152:0/2855111546 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4e0c013f00 con 0x7f4e10060280 2023-12-06T03:55:58.141 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.137+0000 7f4e06ffd700 1 -- 172.21.15.152:0/2855111546 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4e0c014dc0 con 0x7f4e10060280 2023-12-06T03:55:58.141 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.137+0000 7f4e185b9700 1 -- 172.21.15.152:0/2855111546 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e10107260 con 0x7f4e10060280 2023-12-06T03:55:58.142 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.138+0000 7f4e06ffd700 1 -- 172.21.15.152:0/2855111546 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4e0c0095f0 con 0x7f4e10060280 2023-12-06T03:55:58.142 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.138+0000 7f4e185b9700 1 -- 172.21.15.152:0/2855111546 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e10045ad0 con 0x7f4e10060280 2023-12-06T03:55:58.142 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.139+0000 7f4e06ffd700 1 --2- 172.21.15.152:0/2855111546 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4dfc02ff60 0x7f4dfc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:55:58.142 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.139+0000 7f4e06ffd700 1 -- 172.21.15.152:0/2855111546 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4e0c041040 con 0x7f4e10060280 2023-12-06T03:55:58.142 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.139+0000 7f4e15b54700 1 --2- 172.21.15.152:0/2855111546 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4dfc02ff60 0x7f4dfc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:55:58.142 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.139+0000 7f4e15b54700 1 --2- 172.21.15.152:0/2855111546 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4dfc02ff60 0x7f4dfc032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4e000096a0 tx=0x7f4e00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:55:58.145 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.143+0000 7f4e06ffd700 1 -- 172.21.15.152:0/2855111546 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4e0c014420 con 0x7f4e10060280 2023-12-06T03:55:58.351 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.350+0000 7f4e185b9700 1 -- 172.21.15.152:0/2855111546 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4e10108110 con 0x7f4e10060280 2023-12-06T03:55:58.352 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.350+0000 7f4e06ffd700 1 -- 172.21.15.152:0/2855111546 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4e0c010230 con 0x7f4e10060280 2023-12-06T03:55:58.352 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:55:58.352 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:55:58.353 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.352+0000 7f4e185b9700 1 -- 172.21.15.152:0/2855111546 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4dfc02ff60 msgr2=0x7f4dfc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:58.354 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.352+0000 7f4e185b9700 1 --2- 172.21.15.152:0/2855111546 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4dfc02ff60 0x7f4dfc032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f4e000096a0 tx=0x7f4e00006b40).stop 2023-12-06T03:55:58.354 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.352+0000 7f4e185b9700 1 -- 172.21.15.152:0/2855111546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e10060280 msgr2=0x7f4e101066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:55:58.354 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.352+0000 7f4e185b9700 1 --2- 172.21.15.152:0/2855111546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e10060280 0x7f4e101066e0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f4e0c013510 tx=0x7f4e0c00b040).stop 2023-12-06T03:55:58.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.352+0000 7f4e185b9700 1 -- 172.21.15.152:0/2855111546 shutdown_connections 2023-12-06T03:55:58.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.352+0000 7f4e185b9700 1 --2- 172.21.15.152:0/2855111546 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4dfc02ff60 0x7f4dfc032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:58.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.352+0000 7f4e185b9700 1 --2- 172.21.15.152:0/2855111546 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4e10060280 0x7f4e101066e0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:55:58.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.352+0000 7f4e185b9700 1 -- 172.21.15.152:0/2855111546 >> 172.21.15.152:0/2855111546 conn(0x7f4e100f0b70 msgr2=0x7f4e100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:55:58.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.352+0000 7f4e185b9700 1 -- 172.21.15.152:0/2855111546 shutdown_connections 2023-12-06T03:55:58.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:55:58.352+0000 7f4e185b9700 1 -- 172.21.15.152:0/2855111546 wait complete. 2023-12-06T03:55:58.355 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:55:58.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:58 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:55:58.351749+0000 mon.smithi132 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.152:0/2855111546' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:55:59.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:55:59.722 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:55:59.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:55:59 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:55:58.410883+0000 mgr.smithi132.gokxja (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:00.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.869+0000 7fe0f87bb700 1 -- 172.21.15.152:0/708121989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0f00f5410 msgr2=0x7fe0f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:00.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.869+0000 7fe0f87bb700 1 --2- 172.21.15.152:0/708121989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0f00f5410 0x7fe0f00f5830 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fe0e0009fb0 tx=0x7fe0e0009b30).stop 2023-12-06T03:56:00.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.869+0000 7fe0f87bb700 1 -- 172.21.15.152:0/708121989 shutdown_connections 2023-12-06T03:56:00.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.869+0000 7fe0f87bb700 1 --2- 172.21.15.152:0/708121989 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0f00f5410 0x7fe0f00f5830 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:00.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.869+0000 7fe0f87bb700 1 -- 172.21.15.152:0/708121989 >> 172.21.15.152:0/708121989 conn(0x7fe0f00f0b70 msgr2=0x7fe0f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:00.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.869+0000 7fe0f87bb700 1 -- 172.21.15.152:0/708121989 shutdown_connections 2023-12-06T03:56:00.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.869+0000 7fe0f87bb700 1 -- 172.21.15.152:0/708121989 wait complete. 2023-12-06T03:56:00.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.870+0000 7fe0f87bb700 1 Processor -- start 2023-12-06T03:56:00.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.870+0000 7fe0f87bb700 1 -- start start 2023-12-06T03:56:00.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.870+0000 7fe0f87bb700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0f00f5410 0x7fe0f0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:00.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.870+0000 7fe0f87bb700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f0060240 con 0x7fe0f00f5410 2023-12-06T03:56:00.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.871+0000 7fe0f6557700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0f00f5410 0x7fe0f0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:00.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.871+0000 7fe0f6557700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0f00f5410 0x7fe0f0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:60090/0 (socket says 172.21.15.152:60090) 2023-12-06T03:56:00.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.871+0000 7fe0f6557700 1 -- 172.21.15.152:0/1942112050 learned_addr learned my addr 172.21.15.152:0/1942112050 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:00.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.871+0000 7fe0f6557700 1 -- 172.21.15.152:0/1942112050 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0e0005040 con 0x7fe0f00f5410 2023-12-06T03:56:00.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.871+0000 7fe0f6557700 1 --2- 172.21.15.152:0/1942112050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0f00f5410 0x7fe0f0109430 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fe0e0013510 tx=0x7fe0e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:00.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.871+0000 7fe0e6ffd700 1 -- 172.21.15.152:0/1942112050 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0e0009320 con 0x7fe0f00f5410 2023-12-06T03:56:00.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.871+0000 7fe0f87bb700 1 -- 172.21.15.152:0/1942112050 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f0060440 con 0x7fe0f00f5410 2023-12-06T03:56:00.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.871+0000 7fe0e6ffd700 1 -- 172.21.15.152:0/1942112050 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe0e0013f00 con 0x7fe0f00f5410 2023-12-06T03:56:00.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.871+0000 7fe0e6ffd700 1 -- 172.21.15.152:0/1942112050 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0e0014d90 con 0x7fe0f00f5410 2023-12-06T03:56:00.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.871+0000 7fe0f87bb700 1 -- 172.21.15.152:0/1942112050 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f0109ae0 con 0x7fe0f00f5410 2023-12-06T03:56:00.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.872+0000 7fe0e6ffd700 1 -- 172.21.15.152:0/1942112050 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe0e0014420 con 0x7fe0f00f5410 2023-12-06T03:56:00.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.872+0000 7fe0f87bb700 1 -- 172.21.15.152:0/1942112050 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0f0045ad0 con 0x7fe0f00f5410 2023-12-06T03:56:00.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.872+0000 7fe0e6ffd700 1 --2- 172.21.15.152:0/1942112050 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe0dc02ff60 0x7fe0dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:00.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.872+0000 7fe0e6ffd700 1 -- 172.21.15.152:0/1942112050 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe0e0040fb0 con 0x7fe0f00f5410 2023-12-06T03:56:00.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.872+0000 7fe0f5d56700 1 --2- 172.21.15.152:0/1942112050 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe0dc02ff60 0x7fe0dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:00.875 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.873+0000 7fe0f5d56700 1 --2- 172.21.15.152:0/1942112050 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe0dc02ff60 0x7fe0dc032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fe0ec0096a0 tx=0x7fe0ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:00.878 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:00.877+0000 7fe0e6ffd700 1 -- 172.21.15.152:0/1942112050 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe0e00095f0 con 0x7fe0f00f5410 2023-12-06T03:56:01.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:01.082+0000 7fe0f87bb700 1 -- 172.21.15.152:0/1942112050 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe0f00fae60 con 0x7fe0f00f5410 2023-12-06T03:56:01.084 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:01.083+0000 7fe0e6ffd700 1 -- 172.21.15.152:0/1942112050 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe0e001db70 con 0x7fe0f00f5410 2023-12-06T03:56:01.084 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:01.085 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:01.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:01.084+0000 7fe0f87bb700 1 -- 172.21.15.152:0/1942112050 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe0dc02ff60 msgr2=0x7fe0dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:01.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:01.084+0000 7fe0f87bb700 1 --2- 172.21.15.152:0/1942112050 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe0dc02ff60 0x7fe0dc032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fe0ec0096a0 tx=0x7fe0ec006b40).stop 2023-12-06T03:56:01.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:01.084+0000 7fe0f87bb700 1 -- 172.21.15.152:0/1942112050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0f00f5410 msgr2=0x7fe0f0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:01.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:01.084+0000 7fe0f87bb700 1 --2- 172.21.15.152:0/1942112050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0f00f5410 0x7fe0f0109430 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fe0e0013510 tx=0x7fe0e000b040).stop 2023-12-06T03:56:01.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:01.084+0000 7fe0f87bb700 1 -- 172.21.15.152:0/1942112050 shutdown_connections 2023-12-06T03:56:01.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:01.084+0000 7fe0f87bb700 1 --2- 172.21.15.152:0/1942112050 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe0dc02ff60 0x7fe0dc032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:01.087 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:01.084+0000 7fe0f87bb700 1 --2- 172.21.15.152:0/1942112050 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe0f00f5410 0x7fe0f0109430 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:01.087 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:01.084+0000 7fe0f87bb700 1 -- 172.21.15.152:0/1942112050 >> 172.21.15.152:0/1942112050 conn(0x7fe0f00f0b70 msgr2=0x7fe0f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:01.087 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:01.084+0000 7fe0f87bb700 1 -- 172.21.15.152:0/1942112050 shutdown_connections 2023-12-06T03:56:01.087 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:01.084+0000 7fe0f87bb700 1 -- 172.21.15.152:0/1942112050 wait complete. 2023-12-06T03:56:01.087 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:01.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:01 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:00.411246+0000 mgr.smithi132.gokxja (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:01.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:01 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:01.083834+0000 mon.smithi132 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.152:0/1942112050' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:02.477 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:02.477 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:03.579 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.577+0000 7f8a51c08700 1 -- 172.21.15.152:0/2066821016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c0f5410 msgr2=0x7f8a4c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:03.580 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.577+0000 7f8a51c08700 1 --2- 172.21.15.152:0/2066821016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c0f5410 0x7f8a4c0f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f8a3c009fb0 tx=0x7f8a3c009b30).stop 2023-12-06T03:56:03.580 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.577+0000 7f8a51c08700 1 -- 172.21.15.152:0/2066821016 shutdown_connections 2023-12-06T03:56:03.580 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.577+0000 7f8a51c08700 1 --2- 172.21.15.152:0/2066821016 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c0f5410 0x7f8a4c0f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:03.580 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.577+0000 7f8a51c08700 1 -- 172.21.15.152:0/2066821016 >> 172.21.15.152:0/2066821016 conn(0x7f8a4c0f0b70 msgr2=0x7f8a4c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:03.580 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.577+0000 7f8a51c08700 1 -- 172.21.15.152:0/2066821016 shutdown_connections 2023-12-06T03:56:03.580 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.577+0000 7f8a51c08700 1 -- 172.21.15.152:0/2066821016 wait complete. 2023-12-06T03:56:03.580 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.578+0000 7f8a51c08700 1 Processor -- start 2023-12-06T03:56:03.580 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.578+0000 7f8a51c08700 1 -- start start 2023-12-06T03:56:03.580 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.578+0000 7f8a51c08700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c060200 0x7f8a4c109560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:03.580 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.578+0000 7f8a51c08700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a4c109a50 con 0x7f8a4c060200 2023-12-06T03:56:03.582 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.578+0000 7f8a4b7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c060200 0x7f8a4c109560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:03.582 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.578+0000 7f8a4b7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c060200 0x7f8a4c109560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:60102/0 (socket says 172.21.15.152:60102) 2023-12-06T03:56:03.582 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.578+0000 7f8a4b7fe700 1 -- 172.21.15.152:0/3011818729 learned_addr learned my addr 172.21.15.152:0/3011818729 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:03.582 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.579+0000 7f8a4b7fe700 1 -- 172.21.15.152:0/3011818729 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a3c005040 con 0x7f8a4c060200 2023-12-06T03:56:03.582 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.579+0000 7f8a4b7fe700 1 --2- 172.21.15.152:0/3011818729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c060200 0x7f8a4c109560 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f8a3c009f80 tx=0x7f8a3c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:03.583 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.579+0000 7f8a48ff9700 1 -- 172.21.15.152:0/3011818729 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a3c0136c0 con 0x7f8a4c060200 2023-12-06T03:56:03.583 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.579+0000 7f8a48ff9700 1 -- 172.21.15.152:0/3011818729 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a3c013e70 con 0x7f8a4c060200 2023-12-06T03:56:03.583 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.579+0000 7f8a51c08700 1 -- 172.21.15.152:0/3011818729 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a4c109b90 con 0x7f8a4c060200 2023-12-06T03:56:03.583 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.579+0000 7f8a48ff9700 1 -- 172.21.15.152:0/3011818729 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a3c014420 con 0x7f8a4c060200 2023-12-06T03:56:03.583 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.579+0000 7f8a51c08700 1 -- 172.21.15.152:0/3011818729 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a4c109fc0 con 0x7f8a4c060200 2023-12-06T03:56:03.583 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.580+0000 7f8a48ff9700 1 -- 172.21.15.152:0/3011818729 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a3c025060 con 0x7f8a4c060200 2023-12-06T03:56:03.583 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.580+0000 7f8a51c08700 1 -- 172.21.15.152:0/3011818729 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a4c045ad0 con 0x7f8a4c060200 2023-12-06T03:56:03.583 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.580+0000 7f8a48ff9700 1 --2- 172.21.15.152:0/3011818729 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8a3802ff60 0x7f8a38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:03.583 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.580+0000 7f8a48ff9700 1 -- 172.21.15.152:0/3011818729 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a3c040f00 con 0x7f8a4c060200 2023-12-06T03:56:03.583 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.580+0000 7f8a43fff700 1 --2- 172.21.15.152:0/3011818729 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8a3802ff60 0x7f8a38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:03.583 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.580+0000 7f8a43fff700 1 --2- 172.21.15.152:0/3011818729 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8a3802ff60 0x7f8a38032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f8a340096a0 tx=0x7f8a34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:03.586 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.585+0000 7f8a48ff9700 1 -- 172.21.15.152:0/3011818729 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a3c01d3a0 con 0x7f8a4c060200 2023-12-06T03:56:03.797 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.796+0000 7f8a51c08700 1 -- 172.21.15.152:0/3011818729 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a4c0fae60 con 0x7f8a4c060200 2023-12-06T03:56:03.798 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.796+0000 7f8a48ff9700 1 -- 172.21.15.152:0/3011818729 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8a3c004490 con 0x7f8a4c060200 2023-12-06T03:56:03.799 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:03.799 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:03.800 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.798+0000 7f8a51c08700 1 -- 172.21.15.152:0/3011818729 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8a3802ff60 msgr2=0x7f8a38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:03.800 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.798+0000 7f8a51c08700 1 --2- 172.21.15.152:0/3011818729 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8a3802ff60 0x7f8a38032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f8a340096a0 tx=0x7f8a34006b40).stop 2023-12-06T03:56:03.800 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.798+0000 7f8a51c08700 1 -- 172.21.15.152:0/3011818729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c060200 msgr2=0x7f8a4c109560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:03.800 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.798+0000 7f8a51c08700 1 --2- 172.21.15.152:0/3011818729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c060200 0x7f8a4c109560 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f8a3c009f80 tx=0x7f8a3c009b30).stop 2023-12-06T03:56:03.800 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.798+0000 7f8a51c08700 1 -- 172.21.15.152:0/3011818729 shutdown_connections 2023-12-06T03:56:03.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.798+0000 7f8a51c08700 1 --2- 172.21.15.152:0/3011818729 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8a3802ff60 0x7f8a38032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:03.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.798+0000 7f8a51c08700 1 --2- 172.21.15.152:0/3011818729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a4c060200 0x7f8a4c109560 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:03.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.798+0000 7f8a51c08700 1 -- 172.21.15.152:0/3011818729 >> 172.21.15.152:0/3011818729 conn(0x7f8a4c0f0b70 msgr2=0x7f8a4c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:03.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.798+0000 7f8a51c08700 1 -- 172.21.15.152:0/3011818729 shutdown_connections 2023-12-06T03:56:03.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:03.798+0000 7f8a51c08700 1 -- 172.21.15.152:0/3011818729 wait complete. 2023-12-06T03:56:03.801 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:03.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:02.411642+0000 mgr.smithi132.gokxja (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:04.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:04 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:03.797971+0000 mon.smithi132 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.152:0/3011818729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:05.164 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:05.164 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:05.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:04.411968+0000 mgr.smithi132.gokxja (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:06.222 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.218+0000 7fea7eb0b700 1 -- 172.21.15.152:0/3122252096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea780f62a0 msgr2=0x7fea780f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:06.222 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.218+0000 7fea7eb0b700 1 --2- 172.21.15.152:0/3122252096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea780f62a0 0x7fea780f66c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fea6c009fb0 tx=0x7fea6c009b30).stop 2023-12-06T03:56:06.222 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.218+0000 7fea7eb0b700 1 -- 172.21.15.152:0/3122252096 shutdown_connections 2023-12-06T03:56:06.222 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.218+0000 7fea7eb0b700 1 --2- 172.21.15.152:0/3122252096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea780f62a0 0x7fea780f66c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:06.222 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.218+0000 7fea7eb0b700 1 -- 172.21.15.152:0/3122252096 >> 172.21.15.152:0/3122252096 conn(0x7fea780f1aa0 msgr2=0x7fea780f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:06.222 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.219+0000 7fea7eb0b700 1 -- 172.21.15.152:0/3122252096 shutdown_connections 2023-12-06T03:56:06.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.219+0000 7fea7eb0b700 1 -- 172.21.15.152:0/3122252096 wait complete. 2023-12-06T03:56:06.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.219+0000 7fea7eb0b700 1 Processor -- start 2023-12-06T03:56:06.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.219+0000 7fea7eb0b700 1 -- start start 2023-12-06T03:56:06.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.219+0000 7fea7eb0b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea780f62a0 0x7fea781064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:06.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.219+0000 7fea7eb0b700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea781069a0 con 0x7fea780f62a0 2023-12-06T03:56:06.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.219+0000 7fea7db09700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea780f62a0 0x7fea781064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:06.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.219+0000 7fea7db09700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea780f62a0 0x7fea781064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:60132/0 (socket says 172.21.15.152:60132) 2023-12-06T03:56:06.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.219+0000 7fea7db09700 1 -- 172.21.15.152:0/2039629559 learned_addr learned my addr 172.21.15.152:0/2039629559 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:06.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.220+0000 7fea7db09700 1 -- 172.21.15.152:0/2039629559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea6c005040 con 0x7fea780f62a0 2023-12-06T03:56:06.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.220+0000 7fea7db09700 1 --2- 172.21.15.152:0/2039629559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea780f62a0 0x7fea781064b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fea6c013510 tx=0x7fea6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:06.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.220+0000 7fea6a7fc700 1 -- 172.21.15.152:0/2039629559 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea6c009320 con 0x7fea780f62a0 2023-12-06T03:56:06.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.220+0000 7fea6a7fc700 1 -- 172.21.15.152:0/2039629559 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea6c013f00 con 0x7fea780f62a0 2023-12-06T03:56:06.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.220+0000 7fea6a7fc700 1 -- 172.21.15.152:0/2039629559 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea6c014dc0 con 0x7fea780f62a0 2023-12-06T03:56:06.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.220+0000 7fea7eb0b700 1 -- 172.21.15.152:0/2039629559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea78106ba0 con 0x7fea780f62a0 2023-12-06T03:56:06.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.220+0000 7fea7eb0b700 1 -- 172.21.15.152:0/2039629559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea78107030 con 0x7fea780f62a0 2023-12-06T03:56:06.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.221+0000 7fea6a7fc700 1 -- 172.21.15.152:0/2039629559 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea6c0095f0 con 0x7fea780f62a0 2023-12-06T03:56:06.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.221+0000 7fea7eb0b700 1 -- 172.21.15.152:0/2039629559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea78045ad0 con 0x7fea780f62a0 2023-12-06T03:56:06.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.221+0000 7fea6a7fc700 1 --2- 172.21.15.152:0/2039629559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fea6402ffb0 0x7fea64032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:06.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.221+0000 7fea6a7fc700 1 -- 172.21.15.152:0/2039629559 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea6c0410b0 con 0x7fea780f62a0 2023-12-06T03:56:06.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.221+0000 7fea7d308700 1 --2- 172.21.15.152:0/2039629559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fea6402ffb0 0x7fea64032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:06.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.222+0000 7fea7d308700 1 --2- 172.21.15.152:0/2039629559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fea6402ffb0 0x7fea64032470 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fea740096a0 tx=0x7fea74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:06.227 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.226+0000 7fea6a7fc700 1 -- 172.21.15.152:0/2039629559 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea6c06d2b0 con 0x7fea780f62a0 2023-12-06T03:56:06.434 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.431+0000 7fea7eb0b700 1 -- 172.21.15.152:0/2039629559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea78107ee0 con 0x7fea780f62a0 2023-12-06T03:56:06.434 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.432+0000 7fea6a7fc700 1 -- 172.21.15.152:0/2039629559 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fea6c014420 con 0x7fea780f62a0 2023-12-06T03:56:06.434 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:06.434 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:06.435 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.434+0000 7fea7eb0b700 1 -- 172.21.15.152:0/2039629559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fea6402ffb0 msgr2=0x7fea64032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:06.435 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.434+0000 7fea7eb0b700 1 --2- 172.21.15.152:0/2039629559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fea6402ffb0 0x7fea64032470 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fea740096a0 tx=0x7fea74006b40).stop 2023-12-06T03:56:06.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.434+0000 7fea7eb0b700 1 -- 172.21.15.152:0/2039629559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea780f62a0 msgr2=0x7fea781064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:06.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.434+0000 7fea7eb0b700 1 --2- 172.21.15.152:0/2039629559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea780f62a0 0x7fea781064b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fea6c013510 tx=0x7fea6c00b040).stop 2023-12-06T03:56:06.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.434+0000 7fea7eb0b700 1 -- 172.21.15.152:0/2039629559 shutdown_connections 2023-12-06T03:56:06.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.434+0000 7fea7eb0b700 1 --2- 172.21.15.152:0/2039629559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fea6402ffb0 0x7fea64032470 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:06.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.434+0000 7fea7eb0b700 1 --2- 172.21.15.152:0/2039629559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fea780f62a0 0x7fea781064b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:06.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.434+0000 7fea7eb0b700 1 -- 172.21.15.152:0/2039629559 >> 172.21.15.152:0/2039629559 conn(0x7fea780f1aa0 msgr2=0x7fea780f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:06.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.434+0000 7fea7eb0b700 1 -- 172.21.15.152:0/2039629559 shutdown_connections 2023-12-06T03:56:06.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:06.434+0000 7fea7eb0b700 1 -- 172.21.15.152:0/2039629559 wait complete. 2023-12-06T03:56:06.436 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:06.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:06.433444+0000 mon.smithi132 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.152:0/2039629559' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:07.813 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:07.813 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:07.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:07 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:06.412248+0000 mgr.smithi132.gokxja (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:08.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.921+0000 7f8b5823b700 1 -- 172.21.15.152:0/371034373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b50060220 msgr2=0x7f8b500fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:08.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.921+0000 7f8b5823b700 1 --2- 172.21.15.152:0/371034373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b50060220 0x7f8b500fb550 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f8b40009fb0 tx=0x7f8b40009b30).stop 2023-12-06T03:56:08.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.921+0000 7f8b5823b700 1 -- 172.21.15.152:0/371034373 shutdown_connections 2023-12-06T03:56:08.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.921+0000 7f8b5823b700 1 --2- 172.21.15.152:0/371034373 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b50060220 0x7f8b500fb550 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:08.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.921+0000 7f8b5823b700 1 -- 172.21.15.152:0/371034373 >> 172.21.15.152:0/371034373 conn(0x7f8b500f0b70 msgr2=0x7f8b500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:08.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.921+0000 7f8b5823b700 1 -- 172.21.15.152:0/371034373 shutdown_connections 2023-12-06T03:56:08.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.921+0000 7f8b5823b700 1 -- 172.21.15.152:0/371034373 wait complete. 2023-12-06T03:56:08.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.922+0000 7f8b5823b700 1 Processor -- start 2023-12-06T03:56:08.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.922+0000 7f8b5823b700 1 -- start start 2023-12-06T03:56:08.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.922+0000 7f8b5823b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b50060220 0x7f8b501088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:08.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.922+0000 7f8b5823b700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b50108db0 con 0x7f8b50060220 2023-12-06T03:56:08.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.922+0000 7f8b55fd7700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b50060220 0x7f8b501088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:08.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.922+0000 7f8b55fd7700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b50060220 0x7f8b501088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:60150/0 (socket says 172.21.15.152:60150) 2023-12-06T03:56:08.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.922+0000 7f8b55fd7700 1 -- 172.21.15.152:0/430552899 learned_addr learned my addr 172.21.15.152:0/430552899 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:08.928 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.923+0000 7f8b55fd7700 1 -- 172.21.15.152:0/430552899 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b40005040 con 0x7f8b50060220 2023-12-06T03:56:08.929 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.923+0000 7f8b55fd7700 1 --2- 172.21.15.152:0/430552899 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b50060220 0x7f8b501088c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f8b40013510 tx=0x7f8b4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:08.929 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.923+0000 7f8b467fc700 1 -- 172.21.15.152:0/430552899 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b40009320 con 0x7f8b50060220 2023-12-06T03:56:08.929 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.923+0000 7f8b467fc700 1 -- 172.21.15.152:0/430552899 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8b40013f00 con 0x7f8b50060220 2023-12-06T03:56:08.929 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.923+0000 7f8b5823b700 1 -- 172.21.15.152:0/430552899 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b50108fb0 con 0x7f8b50060220 2023-12-06T03:56:08.929 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.923+0000 7f8b467fc700 1 -- 172.21.15.152:0/430552899 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8b40014dc0 con 0x7f8b50060220 2023-12-06T03:56:08.929 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.923+0000 7f8b5823b700 1 -- 172.21.15.152:0/430552899 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b50109440 con 0x7f8b50060220 2023-12-06T03:56:08.929 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.924+0000 7f8b467fc700 1 -- 172.21.15.152:0/430552899 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8b40014420 con 0x7f8b50060220 2023-12-06T03:56:08.929 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.924+0000 7f8b5823b700 1 -- 172.21.15.152:0/430552899 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b50045ad0 con 0x7f8b50060220 2023-12-06T03:56:08.929 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.924+0000 7f8b467fc700 1 --2- 172.21.15.152:0/430552899 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8b3c02ff60 0x7f8b3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:08.929 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.924+0000 7f8b467fc700 1 -- 172.21.15.152:0/430552899 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8b40041110 con 0x7f8b50060220 2023-12-06T03:56:08.929 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.924+0000 7f8b557d6700 1 --2- 172.21.15.152:0/430552899 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8b3c02ff60 0x7f8b3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:08.929 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.924+0000 7f8b557d6700 1 --2- 172.21.15.152:0/430552899 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8b3c02ff60 0x7f8b3c032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f8b4c0096a0 tx=0x7f8b4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:08.930 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:08.928+0000 7f8b467fc700 1 -- 172.21.15.152:0/430552899 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8b400095f0 con 0x7f8b50060220 2023-12-06T03:56:09.138 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:09.137+0000 7f8b5823b700 1 -- 172.21.15.152:0/430552899 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8b5010a2f0 con 0x7f8b50060220 2023-12-06T03:56:09.138 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:09.138+0000 7f8b467fc700 1 -- 172.21.15.152:0/430552899 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8b4001cb10 con 0x7f8b50060220 2023-12-06T03:56:09.139 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:09.139 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:09.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:09.139+0000 7f8b5823b700 1 -- 172.21.15.152:0/430552899 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8b3c02ff60 msgr2=0x7f8b3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:09.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:09.139+0000 7f8b5823b700 1 --2- 172.21.15.152:0/430552899 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8b3c02ff60 0x7f8b3c032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f8b4c0096a0 tx=0x7f8b4c006b40).stop 2023-12-06T03:56:09.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:09.139+0000 7f8b5823b700 1 -- 172.21.15.152:0/430552899 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b50060220 msgr2=0x7f8b501088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:09.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:09.139+0000 7f8b5823b700 1 --2- 172.21.15.152:0/430552899 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b50060220 0x7f8b501088c0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f8b40013510 tx=0x7f8b4000b040).stop 2023-12-06T03:56:09.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:09.139+0000 7f8b5823b700 1 -- 172.21.15.152:0/430552899 shutdown_connections 2023-12-06T03:56:09.140 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:09.139+0000 7f8b5823b700 1 --2- 172.21.15.152:0/430552899 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8b3c02ff60 0x7f8b3c032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:09.141 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:09.139+0000 7f8b5823b700 1 --2- 172.21.15.152:0/430552899 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8b50060220 0x7f8b501088c0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:09.141 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:09.139+0000 7f8b5823b700 1 -- 172.21.15.152:0/430552899 >> 172.21.15.152:0/430552899 conn(0x7f8b500f0b70 msgr2=0x7f8b500f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:09.141 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:09.139+0000 7f8b5823b700 1 -- 172.21.15.152:0/430552899 shutdown_connections 2023-12-06T03:56:09.141 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:09.139+0000 7f8b5823b700 1 -- 172.21.15.152:0/430552899 wait complete. 2023-12-06T03:56:09.141 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:09.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:09 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:08.412581+0000 mgr.smithi132.gokxja (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:09.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:09 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:09.138098+0000 mon.smithi132 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.152:0/430552899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:10.502 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:10.502 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:11.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.660+0000 7f68281f3700 1 -- 172.21.15.152:0/2316068615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68200f5410 msgr2=0x7f68200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:11.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.660+0000 7f68281f3700 1 --2- 172.21.15.152:0/2316068615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68200f5410 0x7f68200f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f6814009fb0 tx=0x7f6814009b30).stop 2023-12-06T03:56:11.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.660+0000 7f68281f3700 1 -- 172.21.15.152:0/2316068615 shutdown_connections 2023-12-06T03:56:11.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.660+0000 7f68281f3700 1 --2- 172.21.15.152:0/2316068615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68200f5410 0x7f68200f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:11.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.660+0000 7f68281f3700 1 -- 172.21.15.152:0/2316068615 >> 172.21.15.152:0/2316068615 conn(0x7f68200f0b70 msgr2=0x7f68200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:11.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.660+0000 7f68281f3700 1 -- 172.21.15.152:0/2316068615 shutdown_connections 2023-12-06T03:56:11.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.660+0000 7f68281f3700 1 -- 172.21.15.152:0/2316068615 wait complete. 2023-12-06T03:56:11.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.661+0000 7f68281f3700 1 Processor -- start 2023-12-06T03:56:11.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.661+0000 7f68281f3700 1 -- start start 2023-12-06T03:56:11.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.661+0000 7f68281f3700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68200f5410 0x7f6820105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:11.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.661+0000 7f68281f3700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6820105b20 con 0x7f68200f5410 2023-12-06T03:56:11.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.661+0000 7f6825f8f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68200f5410 0x7f6820105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:11.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.661+0000 7f6825f8f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68200f5410 0x7f6820105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:34190/0 (socket says 172.21.15.152:34190) 2023-12-06T03:56:11.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.661+0000 7f6825f8f700 1 -- 172.21.15.152:0/1661383785 learned_addr learned my addr 172.21.15.152:0/1661383785 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:11.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.661+0000 7f6825f8f700 1 -- 172.21.15.152:0/1661383785 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6814005040 con 0x7f68200f5410 2023-12-06T03:56:11.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.661+0000 7f6825f8f700 1 --2- 172.21.15.152:0/1661383785 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68200f5410 0x7f6820105630 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f6814013510 tx=0x7f681400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:11.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.662+0000 7f68127fc700 1 -- 172.21.15.152:0/1661383785 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6814009320 con 0x7f68200f5410 2023-12-06T03:56:11.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.662+0000 7f68281f3700 1 -- 172.21.15.152:0/1661383785 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6820105d20 con 0x7f68200f5410 2023-12-06T03:56:11.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.662+0000 7f68127fc700 1 -- 172.21.15.152:0/1661383785 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6814013f00 con 0x7f68200f5410 2023-12-06T03:56:11.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.662+0000 7f68127fc700 1 -- 172.21.15.152:0/1661383785 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6814014d90 con 0x7f68200f5410 2023-12-06T03:56:11.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.662+0000 7f68281f3700 1 -- 172.21.15.152:0/1661383785 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68201061b0 con 0x7f68200f5410 2023-12-06T03:56:11.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.662+0000 7f68127fc700 1 -- 172.21.15.152:0/1661383785 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68140095f0 con 0x7f68200f5410 2023-12-06T03:56:11.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.662+0000 7f68281f3700 1 -- 172.21.15.152:0/1661383785 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6820045ad0 con 0x7f68200f5410 2023-12-06T03:56:11.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.663+0000 7f68127fc700 1 --2- 172.21.15.152:0/1661383785 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f680c02ff60 0x7f680c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:11.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.663+0000 7f68127fc700 1 -- 172.21.15.152:0/1661383785 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6814041000 con 0x7f68200f5410 2023-12-06T03:56:11.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.663+0000 7f682578e700 1 --2- 172.21.15.152:0/1661383785 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f680c02ff60 0x7f680c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:11.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.663+0000 7f682578e700 1 --2- 172.21.15.152:0/1661383785 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f680c02ff60 0x7f680c032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f681c0096a0 tx=0x7f681c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:11.669 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.667+0000 7f68127fc700 1 -- 172.21.15.152:0/1661383785 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6814014420 con 0x7f68200f5410 2023-12-06T03:56:11.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:11 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:10.412967+0000 mgr.smithi132.gokxja (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:11.876 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.876+0000 7f68281f3700 1 -- 172.21.15.152:0/1661383785 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68200fae60 con 0x7f68200f5410 2023-12-06T03:56:11.877 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.876+0000 7f68127fc700 1 -- 172.21.15.152:0/1661383785 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6814010230 con 0x7f68200f5410 2023-12-06T03:56:11.877 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:11.877 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:11.878 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.878+0000 7f68281f3700 1 -- 172.21.15.152:0/1661383785 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f680c02ff60 msgr2=0x7f680c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:11.878 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.878+0000 7f68281f3700 1 --2- 172.21.15.152:0/1661383785 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f680c02ff60 0x7f680c032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f681c0096a0 tx=0x7f681c006b40).stop 2023-12-06T03:56:11.879 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.878+0000 7f68281f3700 1 -- 172.21.15.152:0/1661383785 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68200f5410 msgr2=0x7f6820105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:11.879 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.878+0000 7f68281f3700 1 --2- 172.21.15.152:0/1661383785 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68200f5410 0x7f6820105630 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f6814013510 tx=0x7f681400b040).stop 2023-12-06T03:56:11.879 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.878+0000 7f68281f3700 1 -- 172.21.15.152:0/1661383785 shutdown_connections 2023-12-06T03:56:11.879 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.878+0000 7f68281f3700 1 --2- 172.21.15.152:0/1661383785 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f680c02ff60 0x7f680c032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:11.879 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.878+0000 7f68281f3700 1 --2- 172.21.15.152:0/1661383785 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68200f5410 0x7f6820105630 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:11.879 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.878+0000 7f68281f3700 1 -- 172.21.15.152:0/1661383785 >> 172.21.15.152:0/1661383785 conn(0x7f68200f0b70 msgr2=0x7f68200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:11.879 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.878+0000 7f68281f3700 1 -- 172.21.15.152:0/1661383785 shutdown_connections 2023-12-06T03:56:11.879 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:11.878+0000 7f68281f3700 1 -- 172.21.15.152:0/1661383785 wait complete. 2023-12-06T03:56:11.879 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:12.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:11.876675+0000 mon.smithi132 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.152:0/1661383785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:13.273 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:13.273 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:13.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:13 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:12.413330+0000 mgr.smithi132.gokxja (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:14.396 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.395+0000 7fbbad335700 1 -- 172.21.15.152:0/383457848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbba80f79a0 msgr2=0x7fbba80f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:14.396 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.395+0000 7fbbad335700 1 --2- 172.21.15.152:0/383457848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbba80f79a0 0x7fbba80f7dc0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7fbb98009fb0 tx=0x7fbb98009b30).stop 2023-12-06T03:56:14.396 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.395+0000 7fbbad335700 1 -- 172.21.15.152:0/383457848 shutdown_connections 2023-12-06T03:56:14.397 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.395+0000 7fbbad335700 1 --2- 172.21.15.152:0/383457848 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbba80f79a0 0x7fbba80f7dc0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:14.397 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.396+0000 7fbbad335700 1 -- 172.21.15.152:0/383457848 >> 172.21.15.152:0/383457848 conn(0x7fbba80f0120 msgr2=0x7fbba80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:14.397 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.396+0000 7fbbad335700 1 -- 172.21.15.152:0/383457848 shutdown_connections 2023-12-06T03:56:14.397 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.396+0000 7fbbad335700 1 -- 172.21.15.152:0/383457848 wait complete. 2023-12-06T03:56:14.399 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.396+0000 7fbbad335700 1 Processor -- start 2023-12-06T03:56:14.399 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.396+0000 7fbbad335700 1 -- start start 2023-12-06T03:56:14.400 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.396+0000 7fbbad335700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbba80f79a0 0x7fbba81064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:14.400 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.396+0000 7fbbad335700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba81069b0 con 0x7fbba80f79a0 2023-12-06T03:56:14.400 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.397+0000 7fbba7fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbba80f79a0 0x7fbba81064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:14.400 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.397+0000 7fbba7fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbba80f79a0 0x7fbba81064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:34200/0 (socket says 172.21.15.152:34200) 2023-12-06T03:56:14.400 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.397+0000 7fbba7fff700 1 -- 172.21.15.152:0/590651276 learned_addr learned my addr 172.21.15.152:0/590651276 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:14.400 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.397+0000 7fbba7fff700 1 -- 172.21.15.152:0/590651276 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb98005040 con 0x7fbba80f79a0 2023-12-06T03:56:14.400 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.397+0000 7fbba7fff700 1 --2- 172.21.15.152:0/590651276 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbba80f79a0 0x7fbba81064c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbb98013510 tx=0x7fbb9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:14.400 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.397+0000 7fbba4ff9700 1 -- 172.21.15.152:0/590651276 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb98009320 con 0x7fbba80f79a0 2023-12-06T03:56:14.400 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.397+0000 7fbbad335700 1 -- 172.21.15.152:0/590651276 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba8106bb0 con 0x7fbba80f79a0 2023-12-06T03:56:14.400 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.397+0000 7fbba4ff9700 1 -- 172.21.15.152:0/590651276 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb98013f00 con 0x7fbba80f79a0 2023-12-06T03:56:14.400 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.397+0000 7fbba4ff9700 1 -- 172.21.15.152:0/590651276 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb98014dc0 con 0x7fbba80f79a0 2023-12-06T03:56:14.400 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.397+0000 7fbbad335700 1 -- 172.21.15.152:0/590651276 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba8107040 con 0x7fbba80f79a0 2023-12-06T03:56:14.400 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.398+0000 7fbba4ff9700 1 -- 172.21.15.152:0/590651276 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb980095f0 con 0x7fbba80f79a0 2023-12-06T03:56:14.401 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.398+0000 7fbbad335700 1 -- 172.21.15.152:0/590651276 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba8045ad0 con 0x7fbba80f79a0 2023-12-06T03:56:14.401 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.398+0000 7fbba4ff9700 1 --2- 172.21.15.152:0/590651276 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb9002ff60 0x7fbb90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:14.401 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.398+0000 7fbba4ff9700 1 -- 172.21.15.152:0/590651276 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb980410c0 con 0x7fbba80f79a0 2023-12-06T03:56:14.401 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.398+0000 7fbba77fe700 1 --2- 172.21.15.152:0/590651276 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb9002ff60 0x7fbb90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:14.401 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.399+0000 7fbba77fe700 1 --2- 172.21.15.152:0/590651276 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb9002ff60 0x7fbb90032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbb9c0096a0 tx=0x7fbb9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:14.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.403+0000 7fbba4ff9700 1 -- 172.21.15.152:0/590651276 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb9801c790 con 0x7fbba80f79a0 2023-12-06T03:56:14.610 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.610+0000 7fbbad335700 1 -- 172.21.15.152:0/590651276 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbba8107ef0 con 0x7fbba80f79a0 2023-12-06T03:56:14.611 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.610+0000 7fbba4ff9700 1 -- 172.21.15.152:0/590651276 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbb98010230 con 0x7fbba80f79a0 2023-12-06T03:56:14.611 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:14.611 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:14.613 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.612+0000 7fbbad335700 1 -- 172.21.15.152:0/590651276 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb9002ff60 msgr2=0x7fbb90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:14.613 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.612+0000 7fbbad335700 1 --2- 172.21.15.152:0/590651276 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb9002ff60 0x7fbb90032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fbb9c0096a0 tx=0x7fbb9c006b40).stop 2023-12-06T03:56:14.613 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.612+0000 7fbbad335700 1 -- 172.21.15.152:0/590651276 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbba80f79a0 msgr2=0x7fbba81064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:14.613 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.612+0000 7fbbad335700 1 --2- 172.21.15.152:0/590651276 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbba80f79a0 0x7fbba81064c0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fbb98013510 tx=0x7fbb9800b040).stop 2023-12-06T03:56:14.613 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.612+0000 7fbbad335700 1 -- 172.21.15.152:0/590651276 shutdown_connections 2023-12-06T03:56:14.613 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.612+0000 7fbbad335700 1 --2- 172.21.15.152:0/590651276 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb9002ff60 0x7fbb90032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:14.613 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.612+0000 7fbbad335700 1 --2- 172.21.15.152:0/590651276 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbba80f79a0 0x7fbba81064c0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:14.613 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.612+0000 7fbbad335700 1 -- 172.21.15.152:0/590651276 >> 172.21.15.152:0/590651276 conn(0x7fbba80f0120 msgr2=0x7fbba80f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:14.613 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.612+0000 7fbbad335700 1 -- 172.21.15.152:0/590651276 shutdown_connections 2023-12-06T03:56:14.613 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:14.612+0000 7fbbad335700 1 -- 172.21.15.152:0/590651276 wait complete. 2023-12-06T03:56:14.614 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:15.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:15 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:14.413614+0000 mgr.smithi132.gokxja (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:15.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:15 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:14.610650+0000 mon.smithi132 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.152:0/590651276' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:16.010 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:16.011 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:17.079 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.077+0000 7fbb976af700 1 -- 172.21.15.152:0/1364777489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb900f5410 msgr2=0x7fbb900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:17.079 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.077+0000 7fbb976af700 1 --2- 172.21.15.152:0/1364777489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb900f5410 0x7fbb900f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fbb80009fb0 tx=0x7fbb80009b30).stop 2023-12-06T03:56:17.079 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.077+0000 7fbb976af700 1 -- 172.21.15.152:0/1364777489 shutdown_connections 2023-12-06T03:56:17.079 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.077+0000 7fbb976af700 1 --2- 172.21.15.152:0/1364777489 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb900f5410 0x7fbb900f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:17.079 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.077+0000 7fbb976af700 1 -- 172.21.15.152:0/1364777489 >> 172.21.15.152:0/1364777489 conn(0x7fbb900f0b70 msgr2=0x7fbb900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:17.079 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.077+0000 7fbb976af700 1 -- 172.21.15.152:0/1364777489 shutdown_connections 2023-12-06T03:56:17.079 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.077+0000 7fbb976af700 1 -- 172.21.15.152:0/1364777489 wait complete. 2023-12-06T03:56:17.079 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.078+0000 7fbb976af700 1 Processor -- start 2023-12-06T03:56:17.079 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.078+0000 7fbb976af700 1 -- start start 2023-12-06T03:56:17.079 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.078+0000 7fbb976af700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb900f5410 0x7fbb90105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:17.079 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.078+0000 7fbb976af700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb90105b20 con 0x7fbb900f5410 2023-12-06T03:56:17.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.078+0000 7fbb9544b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb900f5410 0x7fbb90105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:17.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.078+0000 7fbb9544b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb900f5410 0x7fbb90105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:34216/0 (socket says 172.21.15.152:34216) 2023-12-06T03:56:17.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.078+0000 7fbb9544b700 1 -- 172.21.15.152:0/23914737 learned_addr learned my addr 172.21.15.152:0/23914737 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:17.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.078+0000 7fbb9544b700 1 -- 172.21.15.152:0/23914737 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb80005040 con 0x7fbb900f5410 2023-12-06T03:56:17.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.078+0000 7fbb9544b700 1 --2- 172.21.15.152:0/23914737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb900f5410 0x7fbb90105630 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fbb80013510 tx=0x7fbb8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:17.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.079+0000 7fbb85ffb700 1 -- 172.21.15.152:0/23914737 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb80009320 con 0x7fbb900f5410 2023-12-06T03:56:17.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.079+0000 7fbb85ffb700 1 -- 172.21.15.152:0/23914737 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb80013f00 con 0x7fbb900f5410 2023-12-06T03:56:17.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.079+0000 7fbb976af700 1 -- 172.21.15.152:0/23914737 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb90105d20 con 0x7fbb900f5410 2023-12-06T03:56:17.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.079+0000 7fbb85ffb700 1 -- 172.21.15.152:0/23914737 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb80014d90 con 0x7fbb900f5410 2023-12-06T03:56:17.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.079+0000 7fbb976af700 1 -- 172.21.15.152:0/23914737 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb901061b0 con 0x7fbb900f5410 2023-12-06T03:56:17.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.080+0000 7fbb976af700 1 -- 172.21.15.152:0/23914737 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb90045ad0 con 0x7fbb900f5410 2023-12-06T03:56:17.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.080+0000 7fbb85ffb700 1 -- 172.21.15.152:0/23914737 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb800095f0 con 0x7fbb900f5410 2023-12-06T03:56:17.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.080+0000 7fbb85ffb700 1 --2- 172.21.15.152:0/23914737 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb7c02ff60 0x7fbb7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:17.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.080+0000 7fbb85ffb700 1 -- 172.21.15.152:0/23914737 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb80040fe0 con 0x7fbb900f5410 2023-12-06T03:56:17.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.080+0000 7fbb94c4a700 1 --2- 172.21.15.152:0/23914737 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb7c02ff60 0x7fbb7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:17.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.081+0000 7fbb94c4a700 1 --2- 172.21.15.152:0/23914737 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb7c02ff60 0x7fbb7c032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbb8c0096a0 tx=0x7fbb8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:17.085 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.085+0000 7fbb85ffb700 1 -- 172.21.15.152:0/23914737 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb80014420 con 0x7fbb900f5410 2023-12-06T03:56:17.290 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.289+0000 7fbb976af700 1 -- 172.21.15.152:0/23914737 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb900fae60 con 0x7fbb900f5410 2023-12-06T03:56:17.290 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.290+0000 7fbb85ffb700 1 -- 172.21.15.152:0/23914737 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbb80010230 con 0x7fbb900f5410 2023-12-06T03:56:17.291 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:17.291 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:17.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.291+0000 7fbb976af700 1 -- 172.21.15.152:0/23914737 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb7c02ff60 msgr2=0x7fbb7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:17.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.291+0000 7fbb976af700 1 --2- 172.21.15.152:0/23914737 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb7c02ff60 0x7fbb7c032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fbb8c0096a0 tx=0x7fbb8c006b40).stop 2023-12-06T03:56:17.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.292+0000 7fbb976af700 1 -- 172.21.15.152:0/23914737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb900f5410 msgr2=0x7fbb90105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:17.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.292+0000 7fbb976af700 1 --2- 172.21.15.152:0/23914737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb900f5410 0x7fbb90105630 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fbb80013510 tx=0x7fbb8000b040).stop 2023-12-06T03:56:17.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.292+0000 7fbb976af700 1 -- 172.21.15.152:0/23914737 shutdown_connections 2023-12-06T03:56:17.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.292+0000 7fbb976af700 1 --2- 172.21.15.152:0/23914737 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb7c02ff60 0x7fbb7c032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:17.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.292+0000 7fbb976af700 1 --2- 172.21.15.152:0/23914737 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb900f5410 0x7fbb90105630 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:17.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.292+0000 7fbb976af700 1 -- 172.21.15.152:0/23914737 >> 172.21.15.152:0/23914737 conn(0x7fbb900f0b70 msgr2=0x7fbb900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:17.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.292+0000 7fbb976af700 1 -- 172.21.15.152:0/23914737 shutdown_connections 2023-12-06T03:56:17.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:17.292+0000 7fbb976af700 1 -- 172.21.15.152:0/23914737 wait complete. 2023-12-06T03:56:17.293 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:17.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:17 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:16.413900+0000 mgr.smithi132.gokxja (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:17.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:17 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:17.290343+0000 mon.smithi132 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.152:0/23914737' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:18.662 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:18.662 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:19.755 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.753+0000 7fcd74816700 1 -- 172.21.15.152:0/1069472275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd700f5410 msgr2=0x7fcd700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:19.755 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.753+0000 7fcd74816700 1 --2- 172.21.15.152:0/1069472275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd700f5410 0x7fcd700f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fcd60009fb0 tx=0x7fcd60009b30).stop 2023-12-06T03:56:19.755 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.753+0000 7fcd74816700 1 -- 172.21.15.152:0/1069472275 shutdown_connections 2023-12-06T03:56:19.755 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.753+0000 7fcd74816700 1 --2- 172.21.15.152:0/1069472275 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd700f5410 0x7fcd700f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:19.755 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.753+0000 7fcd74816700 1 -- 172.21.15.152:0/1069472275 >> 172.21.15.152:0/1069472275 conn(0x7fcd700f0b70 msgr2=0x7fcd700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:19.755 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.753+0000 7fcd74816700 1 -- 172.21.15.152:0/1069472275 shutdown_connections 2023-12-06T03:56:19.755 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.753+0000 7fcd74816700 1 -- 172.21.15.152:0/1069472275 wait complete. 2023-12-06T03:56:19.755 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.754+0000 7fcd74816700 1 Processor -- start 2023-12-06T03:56:19.755 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.754+0000 7fcd74816700 1 -- start start 2023-12-06T03:56:19.755 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.754+0000 7fcd74816700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd700f5410 0x7fcd70109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:19.755 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.754+0000 7fcd74816700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd70060220 con 0x7fcd700f5410 2023-12-06T03:56:19.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.754+0000 7fcd6e59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd700f5410 0x7fcd70109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:19.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.754+0000 7fcd6e59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd700f5410 0x7fcd70109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:46364/0 (socket says 172.21.15.152:46364) 2023-12-06T03:56:19.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.754+0000 7fcd6e59c700 1 -- 172.21.15.152:0/803109414 learned_addr learned my addr 172.21.15.152:0/803109414 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:19.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.755+0000 7fcd6e59c700 1 -- 172.21.15.152:0/803109414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd60005040 con 0x7fcd700f5410 2023-12-06T03:56:19.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.755+0000 7fcd6e59c700 1 --2- 172.21.15.152:0/803109414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd700f5410 0x7fcd70109450 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fcd60013510 tx=0x7fcd6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:19.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.755+0000 7fcd677fe700 1 -- 172.21.15.152:0/803109414 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd60009320 con 0x7fcd700f5410 2023-12-06T03:56:19.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.755+0000 7fcd677fe700 1 -- 172.21.15.152:0/803109414 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd60013f00 con 0x7fcd700f5410 2023-12-06T03:56:19.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.755+0000 7fcd74816700 1 -- 172.21.15.152:0/803109414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd70060420 con 0x7fcd700f5410 2023-12-06T03:56:19.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.755+0000 7fcd677fe700 1 -- 172.21.15.152:0/803109414 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd60014d90 con 0x7fcd700f5410 2023-12-06T03:56:19.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.755+0000 7fcd74816700 1 -- 172.21.15.152:0/803109414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd70109b00 con 0x7fcd700f5410 2023-12-06T03:56:19.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.756+0000 7fcd677fe700 1 -- 172.21.15.152:0/803109414 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd60014420 con 0x7fcd700f5410 2023-12-06T03:56:19.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.756+0000 7fcd74816700 1 -- 172.21.15.152:0/803109414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd70059100 con 0x7fcd700f5410 2023-12-06T03:56:19.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.756+0000 7fcd677fe700 1 --2- 172.21.15.152:0/803109414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd5c02ffb0 0x7fcd5c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:19.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.756+0000 7fcd677fe700 1 -- 172.21.15.152:0/803109414 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd60041090 con 0x7fcd700f5410 2023-12-06T03:56:19.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.756+0000 7fcd65d9b700 1 --2- 172.21.15.152:0/803109414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd5c02ffb0 0x7fcd5c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:19.759 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.757+0000 7fcd65d9b700 1 --2- 172.21.15.152:0/803109414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd5c02ffb0 0x7fcd5c032470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fcd580096a0 tx=0x7fcd58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:19.761 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.761+0000 7fcd677fe700 1 -- 172.21.15.152:0/803109414 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd6006d2b0 con 0x7fcd700f5410 2023-12-06T03:56:19.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:19 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:18.414155+0000 mgr.smithi132.gokxja (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:19.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.968+0000 7fcd74816700 1 -- 172.21.15.152:0/803109414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd70045ad0 con 0x7fcd700f5410 2023-12-06T03:56:19.969 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.968+0000 7fcd677fe700 1 -- 172.21.15.152:0/803109414 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcd600095f0 con 0x7fcd700f5410 2023-12-06T03:56:19.970 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:19.970 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:19.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.970+0000 7fcd74816700 1 -- 172.21.15.152:0/803109414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd5c02ffb0 msgr2=0x7fcd5c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:19.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.970+0000 7fcd74816700 1 --2- 172.21.15.152:0/803109414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd5c02ffb0 0x7fcd5c032470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fcd580096a0 tx=0x7fcd58006b40).stop 2023-12-06T03:56:19.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.970+0000 7fcd74816700 1 -- 172.21.15.152:0/803109414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd700f5410 msgr2=0x7fcd70109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:19.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.970+0000 7fcd74816700 1 --2- 172.21.15.152:0/803109414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd700f5410 0x7fcd70109450 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fcd60013510 tx=0x7fcd6000b040).stop 2023-12-06T03:56:19.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.970+0000 7fcd74816700 1 -- 172.21.15.152:0/803109414 shutdown_connections 2023-12-06T03:56:19.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.970+0000 7fcd74816700 1 --2- 172.21.15.152:0/803109414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd5c02ffb0 0x7fcd5c032470 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:19.972 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.970+0000 7fcd74816700 1 --2- 172.21.15.152:0/803109414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd700f5410 0x7fcd70109450 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:19.972 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.970+0000 7fcd74816700 1 -- 172.21.15.152:0/803109414 >> 172.21.15.152:0/803109414 conn(0x7fcd700f0b70 msgr2=0x7fcd700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:19.972 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.970+0000 7fcd74816700 1 -- 172.21.15.152:0/803109414 shutdown_connections 2023-12-06T03:56:19.972 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:19.970+0000 7fcd74816700 1 -- 172.21.15.152:0/803109414 wait complete. 2023-12-06T03:56:19.972 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:20.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:19.968805+0000 mon.smithi132 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.152:0/803109414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:21.339 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:21.340 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:21.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:20.414455+0000 mgr.smithi132.gokxja (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:22.467 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.463+0000 7f708cf6f700 1 -- 172.21.15.152:0/3901026976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7088060280 msgr2=0x7f70880f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:22.468 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.463+0000 7f708cf6f700 1 --2- 172.21.15.152:0/3901026976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7088060280 0x7f70880f90f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f707c009fb0 tx=0x7f707c009b30).stop 2023-12-06T03:56:22.468 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.463+0000 7f708cf6f700 1 -- 172.21.15.152:0/3901026976 shutdown_connections 2023-12-06T03:56:22.468 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.463+0000 7f708cf6f700 1 --2- 172.21.15.152:0/3901026976 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7088060280 0x7f70880f90f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:22.468 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.463+0000 7f708cf6f700 1 -- 172.21.15.152:0/3901026976 >> 172.21.15.152:0/3901026976 conn(0x7f70880f0b70 msgr2=0x7f70880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:22.468 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.463+0000 7f708cf6f700 1 -- 172.21.15.152:0/3901026976 shutdown_connections 2023-12-06T03:56:22.468 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.463+0000 7f708cf6f700 1 -- 172.21.15.152:0/3901026976 wait complete. 2023-12-06T03:56:22.468 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.464+0000 7f708cf6f700 1 Processor -- start 2023-12-06T03:56:22.468 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.464+0000 7f708cf6f700 1 -- start start 2023-12-06T03:56:22.468 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.464+0000 7f708cf6f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7088060280 0x7f70881066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:22.468 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.464+0000 7f708cf6f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7088106bd0 con 0x7f7088060280 2023-12-06T03:56:22.468 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.464+0000 7f708659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7088060280 0x7f70881066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:22.468 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.464+0000 7f708659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7088060280 0x7f70881066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:46386/0 (socket says 172.21.15.152:46386) 2023-12-06T03:56:22.469 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.464+0000 7f708659c700 1 -- 172.21.15.152:0/3775096559 learned_addr learned my addr 172.21.15.152:0/3775096559 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:22.469 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.465+0000 7f708659c700 1 -- 172.21.15.152:0/3775096559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707c005040 con 0x7f7088060280 2023-12-06T03:56:22.469 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.465+0000 7f708659c700 1 --2- 172.21.15.152:0/3775096559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7088060280 0x7f70881066e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f707c013510 tx=0x7f707c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:22.469 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.465+0000 7f7076ffd700 1 -- 172.21.15.152:0/3775096559 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f707c009320 con 0x7f7088060280 2023-12-06T03:56:22.469 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.465+0000 7f708cf6f700 1 -- 172.21.15.152:0/3775096559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7088106dd0 con 0x7f7088060280 2023-12-06T03:56:22.469 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.465+0000 7f7076ffd700 1 -- 172.21.15.152:0/3775096559 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f707c013f00 con 0x7f7088060280 2023-12-06T03:56:22.469 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.465+0000 7f7076ffd700 1 -- 172.21.15.152:0/3775096559 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f707c014dc0 con 0x7f7088060280 2023-12-06T03:56:22.469 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.465+0000 7f708cf6f700 1 -- 172.21.15.152:0/3775096559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7088107260 con 0x7f7088060280 2023-12-06T03:56:22.469 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.466+0000 7f7076ffd700 1 -- 172.21.15.152:0/3775096559 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f707c014420 con 0x7f7088060280 2023-12-06T03:56:22.469 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.466+0000 7f708cf6f700 1 -- 172.21.15.152:0/3775096559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7088045ad0 con 0x7f7088060280 2023-12-06T03:56:22.469 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.466+0000 7f7076ffd700 1 --2- 172.21.15.152:0/3775096559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f707002ff60 0x7f7070032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:22.469 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.466+0000 7f7076ffd700 1 -- 172.21.15.152:0/3775096559 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f707c0410b0 con 0x7f7088060280 2023-12-06T03:56:22.469 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.466+0000 7f7085d9b700 1 --2- 172.21.15.152:0/3775096559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f707002ff60 0x7f7070032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:22.470 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.466+0000 7f7085d9b700 1 --2- 172.21.15.152:0/3775096559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f707002ff60 0x7f7070032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f70780096a0 tx=0x7f7078006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:22.471 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.471+0000 7f7076ffd700 1 -- 172.21.15.152:0/3775096559 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f707c0095f0 con 0x7f7088060280 2023-12-06T03:56:22.678 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.676+0000 7f708cf6f700 1 -- 172.21.15.152:0/3775096559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7088108110 con 0x7f7088060280 2023-12-06T03:56:22.679 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.677+0000 7f7076ffd700 1 -- 172.21.15.152:0/3775096559 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f707c01cb10 con 0x7f7088060280 2023-12-06T03:56:22.679 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:22.679 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:22.680 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.678+0000 7f708cf6f700 1 -- 172.21.15.152:0/3775096559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f707002ff60 msgr2=0x7f7070032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:22.680 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.678+0000 7f708cf6f700 1 --2- 172.21.15.152:0/3775096559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f707002ff60 0x7f7070032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f70780096a0 tx=0x7f7078006b40).stop 2023-12-06T03:56:22.680 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.678+0000 7f708cf6f700 1 -- 172.21.15.152:0/3775096559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7088060280 msgr2=0x7f70881066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:22.680 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.678+0000 7f708cf6f700 1 --2- 172.21.15.152:0/3775096559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7088060280 0x7f70881066e0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f707c013510 tx=0x7f707c00b040).stop 2023-12-06T03:56:22.680 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.678+0000 7f708cf6f700 1 -- 172.21.15.152:0/3775096559 shutdown_connections 2023-12-06T03:56:22.680 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.678+0000 7f708cf6f700 1 --2- 172.21.15.152:0/3775096559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f707002ff60 0x7f7070032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:22.680 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.678+0000 7f708cf6f700 1 --2- 172.21.15.152:0/3775096559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7088060280 0x7f70881066e0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:22.680 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.678+0000 7f708cf6f700 1 -- 172.21.15.152:0/3775096559 >> 172.21.15.152:0/3775096559 conn(0x7f70880f0b70 msgr2=0x7f70880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:22.680 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.678+0000 7f708cf6f700 1 -- 172.21.15.152:0/3775096559 shutdown_connections 2023-12-06T03:56:22.680 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:22.678+0000 7f708cf6f700 1 -- 172.21.15.152:0/3775096559 wait complete. 2023-12-06T03:56:22.680 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:23.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:22.414797+0000 mgr.smithi132.gokxja (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:23.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:22.677007+0000 mon.smithi132 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.152:0/3775096559' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:24.051 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:24.051 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:25.160 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.158+0000 7f3cfad88700 1 -- 172.21.15.152:0/3309992358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf40f6a10 msgr2=0x7f3cf40f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:25.160 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.158+0000 7f3cfad88700 1 --2- 172.21.15.152:0/3309992358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf40f6a10 0x7f3cf40f6e30 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f3ce8009fb0 tx=0x7f3ce8009b30).stop 2023-12-06T03:56:25.160 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.158+0000 7f3cfad88700 1 -- 172.21.15.152:0/3309992358 shutdown_connections 2023-12-06T03:56:25.161 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.158+0000 7f3cfad88700 1 --2- 172.21.15.152:0/3309992358 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf40f6a10 0x7f3cf40f6e30 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:25.161 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.158+0000 7f3cfad88700 1 -- 172.21.15.152:0/3309992358 >> 172.21.15.152:0/3309992358 conn(0x7f3cf40f02e0 msgr2=0x7f3cf40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:25.161 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.158+0000 7f3cfad88700 1 -- 172.21.15.152:0/3309992358 shutdown_connections 2023-12-06T03:56:25.161 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.158+0000 7f3cfad88700 1 -- 172.21.15.152:0/3309992358 wait complete. 2023-12-06T03:56:25.161 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.158+0000 7f3cfad88700 1 Processor -- start 2023-12-06T03:56:25.161 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.158+0000 7f3cfad88700 1 -- start start 2023-12-06T03:56:25.161 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.159+0000 7f3cfad88700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf40f6a10 0x7f3cf41093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:25.161 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.159+0000 7f3cfad88700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf40f5330 con 0x7f3cf40f6a10 2023-12-06T03:56:25.161 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.159+0000 7f3cf8b24700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf40f6a10 0x7f3cf41093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:25.161 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.159+0000 7f3cf8b24700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf40f6a10 0x7f3cf41093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:46408/0 (socket says 172.21.15.152:46408) 2023-12-06T03:56:25.161 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.159+0000 7f3cf8b24700 1 -- 172.21.15.152:0/230353010 learned_addr learned my addr 172.21.15.152:0/230353010 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:25.161 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.159+0000 7f3cf8b24700 1 -- 172.21.15.152:0/230353010 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce8005040 con 0x7f3cf40f6a10 2023-12-06T03:56:25.162 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.159+0000 7f3cf8b24700 1 --2- 172.21.15.152:0/230353010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf40f6a10 0x7f3cf41093f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f3ce8013510 tx=0x7f3ce800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:25.162 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.160+0000 7f3cf17fa700 1 -- 172.21.15.152:0/230353010 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce8009320 con 0x7f3cf40f6a10 2023-12-06T03:56:25.162 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.160+0000 7f3cf17fa700 1 -- 172.21.15.152:0/230353010 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ce8013f00 con 0x7f3cf40f6a10 2023-12-06T03:56:25.162 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.160+0000 7f3cfad88700 1 -- 172.21.15.152:0/230353010 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf40f5530 con 0x7f3cf40f6a10 2023-12-06T03:56:25.162 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.160+0000 7f3cf17fa700 1 -- 172.21.15.152:0/230353010 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce8014d90 con 0x7f3cf40f6a10 2023-12-06T03:56:25.162 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.160+0000 7f3cfad88700 1 -- 172.21.15.152:0/230353010 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf4109aa0 con 0x7f3cf40f6a10 2023-12-06T03:56:25.163 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.160+0000 7f3cf17fa700 1 -- 172.21.15.152:0/230353010 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ce80095f0 con 0x7f3cf40f6a10 2023-12-06T03:56:25.163 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.161+0000 7f3cfad88700 1 -- 172.21.15.152:0/230353010 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf4059100 con 0x7f3cf40f6a10 2023-12-06T03:56:25.163 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.161+0000 7f3cf17fa700 1 --2- 172.21.15.152:0/230353010 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce402ff60 0x7f3ce4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:25.163 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.161+0000 7f3cf17fa700 1 -- 172.21.15.152:0/230353010 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ce80410c0 con 0x7f3cf40f6a10 2023-12-06T03:56:25.163 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.161+0000 7f3cf3fff700 1 --2- 172.21.15.152:0/230353010 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce402ff60 0x7f3ce4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:25.163 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.161+0000 7f3cf3fff700 1 --2- 172.21.15.152:0/230353010 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce402ff60 0x7f3ce4032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f3ce00096a0 tx=0x7f3ce0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:25.166 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.165+0000 7f3cf17fa700 1 -- 172.21.15.152:0/230353010 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ce801dbe0 con 0x7f3cf40f6a10 2023-12-06T03:56:25.377 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.376+0000 7f3cfad88700 1 -- 172.21.15.152:0/230353010 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cf4045ad0 con 0x7f3cf40f6a10 2023-12-06T03:56:25.377 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.377+0000 7f3cf17fa700 1 -- 172.21.15.152:0/230353010 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ce801d7f0 con 0x7f3cf40f6a10 2023-12-06T03:56:25.377 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:25.378 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:25.379 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.378+0000 7f3cfad88700 1 -- 172.21.15.152:0/230353010 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce402ff60 msgr2=0x7f3ce4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:25.379 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.378+0000 7f3cfad88700 1 --2- 172.21.15.152:0/230353010 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce402ff60 0x7f3ce4032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f3ce00096a0 tx=0x7f3ce0006b40).stop 2023-12-06T03:56:25.379 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.378+0000 7f3cfad88700 1 -- 172.21.15.152:0/230353010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf40f6a10 msgr2=0x7f3cf41093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:25.379 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.378+0000 7f3cfad88700 1 --2- 172.21.15.152:0/230353010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf40f6a10 0x7f3cf41093f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f3ce8013510 tx=0x7f3ce800b040).stop 2023-12-06T03:56:25.379 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.378+0000 7f3cfad88700 1 -- 172.21.15.152:0/230353010 shutdown_connections 2023-12-06T03:56:25.379 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.378+0000 7f3cfad88700 1 --2- 172.21.15.152:0/230353010 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce402ff60 0x7f3ce4032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:25.379 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.378+0000 7f3cfad88700 1 --2- 172.21.15.152:0/230353010 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf40f6a10 0x7f3cf41093f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:25.379 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.378+0000 7f3cfad88700 1 -- 172.21.15.152:0/230353010 >> 172.21.15.152:0/230353010 conn(0x7f3cf40f02e0 msgr2=0x7f3cf40f0da0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:25.379 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.378+0000 7f3cfad88700 1 -- 172.21.15.152:0/230353010 shutdown_connections 2023-12-06T03:56:25.379 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:25.378+0000 7f3cfad88700 1 -- 172.21.15.152:0/230353010 wait complete. 2023-12-06T03:56:25.380 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:25.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:24.415173+0000 mgr.smithi132.gokxja (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:25.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:25.377026+0000 mon.smithi132 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.152:0/230353010' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:26.765 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:26.766 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:27.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:26.415540+0000 mgr.smithi132.gokxja (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:27.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.869+0000 7f3cf7fff700 1 -- 172.21.15.152:0/142471211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf80f78e0 msgr2=0x7f3cf80f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:27.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.869+0000 7f3cf7fff700 1 --2- 172.21.15.152:0/142471211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf80f78e0 0x7f3cf80f7d00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f3ce8009fb0 tx=0x7f3ce8009b30).stop 2023-12-06T03:56:27.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.869+0000 7f3cf7fff700 1 -- 172.21.15.152:0/142471211 shutdown_connections 2023-12-06T03:56:27.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.869+0000 7f3cf7fff700 1 --2- 172.21.15.152:0/142471211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf80f78e0 0x7f3cf80f7d00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:27.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.869+0000 7f3cf7fff700 1 -- 172.21.15.152:0/142471211 >> 172.21.15.152:0/142471211 conn(0x7f3cf80f0120 msgr2=0x7f3cf80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:27.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.869+0000 7f3cf7fff700 1 -- 172.21.15.152:0/142471211 shutdown_connections 2023-12-06T03:56:27.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.869+0000 7f3cf7fff700 1 -- 172.21.15.152:0/142471211 wait complete. 2023-12-06T03:56:27.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.870+0000 7f3cf7fff700 1 Processor -- start 2023-12-06T03:56:27.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.870+0000 7f3cf7fff700 1 -- start start 2023-12-06T03:56:27.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.870+0000 7f3cf7fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf80f78e0 0x7f3cf81064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:27.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.870+0000 7f3cf7fff700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cf81069d0 con 0x7f3cf80f78e0 2023-12-06T03:56:27.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.870+0000 7f3cf6ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf80f78e0 0x7f3cf81064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:27.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.870+0000 7f3cf6ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf80f78e0 0x7f3cf81064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:46420/0 (socket says 172.21.15.152:46420) 2023-12-06T03:56:27.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.870+0000 7f3cf6ffd700 1 -- 172.21.15.152:0/3932901713 learned_addr learned my addr 172.21.15.152:0/3932901713 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:27.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.870+0000 7f3cf6ffd700 1 -- 172.21.15.152:0/3932901713 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ce8005040 con 0x7f3cf80f78e0 2023-12-06T03:56:27.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.871+0000 7f3cf6ffd700 1 --2- 172.21.15.152:0/3932901713 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf80f78e0 0x7f3cf81064e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f3ce8013510 tx=0x7f3ce800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:27.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.871+0000 7f3cdf7fe700 1 -- 172.21.15.152:0/3932901713 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce8009320 con 0x7f3cf80f78e0 2023-12-06T03:56:27.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.871+0000 7f3cf7fff700 1 -- 172.21.15.152:0/3932901713 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cf8106bd0 con 0x7f3cf80f78e0 2023-12-06T03:56:27.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.871+0000 7f3cdf7fe700 1 -- 172.21.15.152:0/3932901713 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ce8013f00 con 0x7f3cf80f78e0 2023-12-06T03:56:27.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.871+0000 7f3cdf7fe700 1 -- 172.21.15.152:0/3932901713 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ce8014dc0 con 0x7f3cf80f78e0 2023-12-06T03:56:27.872 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.871+0000 7f3cf7fff700 1 -- 172.21.15.152:0/3932901713 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cf8107060 con 0x7f3cf80f78e0 2023-12-06T03:56:27.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.871+0000 7f3cdf7fe700 1 -- 172.21.15.152:0/3932901713 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ce801d3d0 con 0x7f3cf80f78e0 2023-12-06T03:56:27.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.872+0000 7f3cdf7fe700 1 --2- 172.21.15.152:0/3932901713 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce002ff60 0x7f3ce0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:27.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.872+0000 7f3cdf7fe700 1 -- 172.21.15.152:0/3932901713 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ce80410f0 con 0x7f3cf80f78e0 2023-12-06T03:56:27.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.872+0000 7f3cf7fff700 1 -- 172.21.15.152:0/3932901713 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cf8059100 con 0x7f3cf80f78e0 2023-12-06T03:56:27.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.872+0000 7f3cf67fc700 1 --2- 172.21.15.152:0/3932901713 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce002ff60 0x7f3ce0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:27.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.872+0000 7f3cf67fc700 1 --2- 172.21.15.152:0/3932901713 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce002ff60 0x7f3ce0032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f3cec0096a0 tx=0x7f3cec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:27.878 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:27.877+0000 7f3cdf7fe700 1 -- 172.21.15.152:0/3932901713 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ce8012920 con 0x7f3cf80f78e0 2023-12-06T03:56:28.085 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:28.084+0000 7f3cf7fff700 1 -- 172.21.15.152:0/3932901713 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cf80fbce0 con 0x7f3cf80f78e0 2023-12-06T03:56:28.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:28.085+0000 7f3cdf7fe700 1 -- 172.21.15.152:0/3932901713 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ce802b020 con 0x7f3cf80f78e0 2023-12-06T03:56:28.086 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:28.086 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:28.088 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:28.087+0000 7f3cf7fff700 1 -- 172.21.15.152:0/3932901713 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce002ff60 msgr2=0x7f3ce0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:28.088 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:28.087+0000 7f3cf7fff700 1 --2- 172.21.15.152:0/3932901713 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce002ff60 0x7f3ce0032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f3cec0096a0 tx=0x7f3cec006b40).stop 2023-12-06T03:56:28.088 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:28.087+0000 7f3cf7fff700 1 -- 172.21.15.152:0/3932901713 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf80f78e0 msgr2=0x7f3cf81064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:28.088 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:28.087+0000 7f3cf7fff700 1 --2- 172.21.15.152:0/3932901713 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf80f78e0 0x7f3cf81064e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f3ce8013510 tx=0x7f3ce800b040).stop 2023-12-06T03:56:28.088 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:28.087+0000 7f3cf7fff700 1 -- 172.21.15.152:0/3932901713 shutdown_connections 2023-12-06T03:56:28.088 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:28.087+0000 7f3cf7fff700 1 --2- 172.21.15.152:0/3932901713 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f3ce002ff60 0x7f3ce0032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:28.088 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:28.087+0000 7f3cf7fff700 1 --2- 172.21.15.152:0/3932901713 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f3cf80f78e0 0x7f3cf81064e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:28.088 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:28.087+0000 7f3cf7fff700 1 -- 172.21.15.152:0/3932901713 >> 172.21.15.152:0/3932901713 conn(0x7f3cf80f0120 msgr2=0x7f3cf80f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:28.088 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:28.087+0000 7f3cf7fff700 1 -- 172.21.15.152:0/3932901713 shutdown_connections 2023-12-06T03:56:28.088 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:28.087+0000 7f3cf7fff700 1 -- 172.21.15.152:0/3932901713 wait complete. 2023-12-06T03:56:28.088 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:28.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:28 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:28.085564+0000 mon.smithi132 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.152:0/3932901713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:29.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:29.451 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:29.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:28.415827+0000 mgr.smithi132.gokxja (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:30.516 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.514+0000 7f7a5823f700 1 -- 172.21.15.152:0/3818421835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a500f5410 msgr2=0x7f7a500f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:30.516 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.514+0000 7f7a5823f700 1 --2- 172.21.15.152:0/3818421835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a500f5410 0x7f7a500f5830 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f7a40009fb0 tx=0x7f7a40009b30).stop 2023-12-06T03:56:30.517 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.514+0000 7f7a5823f700 1 -- 172.21.15.152:0/3818421835 shutdown_connections 2023-12-06T03:56:30.517 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.514+0000 7f7a5823f700 1 --2- 172.21.15.152:0/3818421835 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a500f5410 0x7f7a500f5830 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:30.517 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.514+0000 7f7a5823f700 1 -- 172.21.15.152:0/3818421835 >> 172.21.15.152:0/3818421835 conn(0x7f7a500f0b70 msgr2=0x7f7a500f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:30.517 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.514+0000 7f7a5823f700 1 -- 172.21.15.152:0/3818421835 shutdown_connections 2023-12-06T03:56:30.517 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.514+0000 7f7a5823f700 1 -- 172.21.15.152:0/3818421835 wait complete. 2023-12-06T03:56:30.519 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.515+0000 7f7a5823f700 1 Processor -- start 2023-12-06T03:56:30.519 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.515+0000 7f7a5823f700 1 -- start start 2023-12-06T03:56:30.519 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.515+0000 7f7a5823f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a500f5410 0x7f7a50106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:30.519 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.515+0000 7f7a5823f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a50106c00 con 0x7f7a500f5410 2023-12-06T03:56:30.519 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.516+0000 7f7a55fdb700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a500f5410 0x7f7a50106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:30.519 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.516+0000 7f7a55fdb700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a500f5410 0x7f7a50106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40034/0 (socket says 172.21.15.152:40034) 2023-12-06T03:56:30.519 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.516+0000 7f7a55fdb700 1 -- 172.21.15.152:0/1288632909 learned_addr learned my addr 172.21.15.152:0/1288632909 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:30.519 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.516+0000 7f7a55fdb700 1 -- 172.21.15.152:0/1288632909 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a40005040 con 0x7f7a500f5410 2023-12-06T03:56:30.519 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.516+0000 7f7a55fdb700 1 --2- 172.21.15.152:0/1288632909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a500f5410 0x7f7a50106710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f7a40014000 tx=0x7f7a4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:30.519 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.516+0000 7f7a467fc700 1 -- 172.21.15.152:0/1288632909 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a400095f0 con 0x7f7a500f5410 2023-12-06T03:56:30.520 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.516+0000 7f7a467fc700 1 -- 172.21.15.152:0/1288632909 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a40007e70 con 0x7f7a500f5410 2023-12-06T03:56:30.520 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.516+0000 7f7a5823f700 1 -- 172.21.15.152:0/1288632909 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a50106e00 con 0x7f7a500f5410 2023-12-06T03:56:30.520 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.516+0000 7f7a467fc700 1 -- 172.21.15.152:0/1288632909 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a40014ea0 con 0x7f7a500f5410 2023-12-06T03:56:30.520 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.516+0000 7f7a5823f700 1 -- 172.21.15.152:0/1288632909 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a50107290 con 0x7f7a500f5410 2023-12-06T03:56:30.520 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.517+0000 7f7a467fc700 1 -- 172.21.15.152:0/1288632909 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a40025070 con 0x7f7a500f5410 2023-12-06T03:56:30.520 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.517+0000 7f7a467fc700 1 --2- 172.21.15.152:0/1288632909 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7a3c02ff60 0x7f7a3c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:30.520 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.517+0000 7f7a5823f700 1 -- 172.21.15.152:0/1288632909 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a50045ad0 con 0x7f7a500f5410 2023-12-06T03:56:30.520 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.517+0000 7f7a467fc700 1 -- 172.21.15.152:0/1288632909 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a40040a30 con 0x7f7a500f5410 2023-12-06T03:56:30.520 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.517+0000 7f7a557da700 1 --2- 172.21.15.152:0/1288632909 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7a3c02ff60 0x7f7a3c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:30.520 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.518+0000 7f7a557da700 1 --2- 172.21.15.152:0/1288632909 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7a3c02ff60 0x7f7a3c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f7a4c0096a0 tx=0x7f7a4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:30.523 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.522+0000 7f7a467fc700 1 -- 172.21.15.152:0/1288632909 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a4000f4e0 con 0x7f7a500f5410 2023-12-06T03:56:30.729 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.727+0000 7f7a5823f700 1 -- 172.21.15.152:0/1288632909 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a500faf50 con 0x7f7a500f5410 2023-12-06T03:56:30.730 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.728+0000 7f7a467fc700 1 -- 172.21.15.152:0/1288632909 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7a40039090 con 0x7f7a500f5410 2023-12-06T03:56:30.730 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:30.730 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:30.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.730+0000 7f7a5823f700 1 -- 172.21.15.152:0/1288632909 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7a3c02ff60 msgr2=0x7f7a3c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:30.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.730+0000 7f7a5823f700 1 --2- 172.21.15.152:0/1288632909 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7a3c02ff60 0x7f7a3c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f7a4c0096a0 tx=0x7f7a4c006b40).stop 2023-12-06T03:56:30.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.730+0000 7f7a5823f700 1 -- 172.21.15.152:0/1288632909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a500f5410 msgr2=0x7f7a50106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:30.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.730+0000 7f7a5823f700 1 --2- 172.21.15.152:0/1288632909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a500f5410 0x7f7a50106710 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f7a40014000 tx=0x7f7a4000b040).stop 2023-12-06T03:56:30.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.730+0000 7f7a5823f700 1 -- 172.21.15.152:0/1288632909 shutdown_connections 2023-12-06T03:56:30.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.730+0000 7f7a5823f700 1 --2- 172.21.15.152:0/1288632909 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7a3c02ff60 0x7f7a3c032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:30.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.730+0000 7f7a5823f700 1 --2- 172.21.15.152:0/1288632909 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7a500f5410 0x7f7a50106710 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:30.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.730+0000 7f7a5823f700 1 -- 172.21.15.152:0/1288632909 >> 172.21.15.152:0/1288632909 conn(0x7f7a500f0b70 msgr2=0x7f7a5005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:30.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.730+0000 7f7a5823f700 1 -- 172.21.15.152:0/1288632909 shutdown_connections 2023-12-06T03:56:30.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:30.730+0000 7f7a5823f700 1 -- 172.21.15.152:0/1288632909 wait complete. 2023-12-06T03:56:30.733 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:31.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:31 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:30.416146+0000 mgr.smithi132.gokxja (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:31.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:31 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:30.729316+0000 mon.smithi132 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.152:0/1288632909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:32.110 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:32.110 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:33.191 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.188+0000 7fc7b066f700 1 -- 172.21.15.152:0/2955657592 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a8060280 msgr2=0x7fc7a80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:33.192 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.188+0000 7fc7b066f700 1 --2- 172.21.15.152:0/2955657592 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a8060280 0x7fc7a80f90f0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fc7a4009fb0 tx=0x7fc7a4009b30).stop 2023-12-06T03:56:33.192 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.188+0000 7fc7b066f700 1 -- 172.21.15.152:0/2955657592 shutdown_connections 2023-12-06T03:56:33.192 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.188+0000 7fc7b066f700 1 --2- 172.21.15.152:0/2955657592 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a8060280 0x7fc7a80f90f0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:33.192 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.188+0000 7fc7b066f700 1 -- 172.21.15.152:0/2955657592 >> 172.21.15.152:0/2955657592 conn(0x7fc7a80f0b70 msgr2=0x7fc7a80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:33.192 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.188+0000 7fc7b066f700 1 -- 172.21.15.152:0/2955657592 shutdown_connections 2023-12-06T03:56:33.192 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.188+0000 7fc7b066f700 1 -- 172.21.15.152:0/2955657592 wait complete. 2023-12-06T03:56:33.192 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.188+0000 7fc7b066f700 1 Processor -- start 2023-12-06T03:56:33.192 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.189+0000 7fc7b066f700 1 -- start start 2023-12-06T03:56:33.192 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.189+0000 7fc7b066f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a8060280 0x7fc7a81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:33.193 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.189+0000 7fc7b066f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7a8106bd0 con 0x7fc7a8060280 2023-12-06T03:56:33.193 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.189+0000 7fc7ae40b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a8060280 0x7fc7a81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:33.193 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.189+0000 7fc7ae40b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a8060280 0x7fc7a81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40046/0 (socket says 172.21.15.152:40046) 2023-12-06T03:56:33.193 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.189+0000 7fc7ae40b700 1 -- 172.21.15.152:0/25063096 learned_addr learned my addr 172.21.15.152:0/25063096 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:33.193 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.189+0000 7fc7ae40b700 1 -- 172.21.15.152:0/25063096 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7a4005040 con 0x7fc7a8060280 2023-12-06T03:56:33.193 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.189+0000 7fc7ae40b700 1 --2- 172.21.15.152:0/25063096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a8060280 0x7fc7a81066e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fc7a4013510 tx=0x7fc7a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:33.193 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.190+0000 7fc79effd700 1 -- 172.21.15.152:0/25063096 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7a4009320 con 0x7fc7a8060280 2023-12-06T03:56:33.193 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.190+0000 7fc7b066f700 1 -- 172.21.15.152:0/25063096 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7a8106dd0 con 0x7fc7a8060280 2023-12-06T03:56:33.193 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.190+0000 7fc79effd700 1 -- 172.21.15.152:0/25063096 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7a4013f00 con 0x7fc7a8060280 2023-12-06T03:56:33.193 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.190+0000 7fc79effd700 1 -- 172.21.15.152:0/25063096 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7a4014dc0 con 0x7fc7a8060280 2023-12-06T03:56:33.193 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.190+0000 7fc7b066f700 1 -- 172.21.15.152:0/25063096 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7a8107260 con 0x7fc7a8060280 2023-12-06T03:56:33.195 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.190+0000 7fc79effd700 1 -- 172.21.15.152:0/25063096 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7a40095f0 con 0x7fc7a8060280 2023-12-06T03:56:33.195 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.191+0000 7fc79effd700 1 --2- 172.21.15.152:0/25063096 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc79402ff60 0x7fc794032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:33.195 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.191+0000 7fc7b066f700 1 -- 172.21.15.152:0/25063096 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7a8045ad0 con 0x7fc7a8060280 2023-12-06T03:56:33.195 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.191+0000 7fc79effd700 1 -- 172.21.15.152:0/25063096 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc7a40410c0 con 0x7fc7a8060280 2023-12-06T03:56:33.195 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.191+0000 7fc7adc0a700 1 --2- 172.21.15.152:0/25063096 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc79402ff60 0x7fc794032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:33.195 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.191+0000 7fc7adc0a700 1 --2- 172.21.15.152:0/25063096 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc79402ff60 0x7fc794032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc7980096a0 tx=0x7fc798006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:33.197 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.195+0000 7fc79effd700 1 -- 172.21.15.152:0/25063096 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc7a401c790 con 0x7fc7a8060280 2023-12-06T03:56:33.402 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.401+0000 7fc7b066f700 1 -- 172.21.15.152:0/25063096 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc7a8108110 con 0x7fc7a8060280 2023-12-06T03:56:33.403 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.401+0000 7fc79effd700 1 -- 172.21.15.152:0/25063096 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc7a402b020 con 0x7fc7a8060280 2023-12-06T03:56:33.403 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:33.403 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:33.405 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.403+0000 7fc7b066f700 1 -- 172.21.15.152:0/25063096 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc79402ff60 msgr2=0x7fc794032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:33.405 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.403+0000 7fc7b066f700 1 --2- 172.21.15.152:0/25063096 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc79402ff60 0x7fc794032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fc7980096a0 tx=0x7fc798006b40).stop 2023-12-06T03:56:33.405 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.403+0000 7fc7b066f700 1 -- 172.21.15.152:0/25063096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a8060280 msgr2=0x7fc7a81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:33.405 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.403+0000 7fc7b066f700 1 --2- 172.21.15.152:0/25063096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a8060280 0x7fc7a81066e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fc7a4013510 tx=0x7fc7a400b040).stop 2023-12-06T03:56:33.405 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.403+0000 7fc7b066f700 1 -- 172.21.15.152:0/25063096 shutdown_connections 2023-12-06T03:56:33.405 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.403+0000 7fc7b066f700 1 --2- 172.21.15.152:0/25063096 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc79402ff60 0x7fc794032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:33.405 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.403+0000 7fc7b066f700 1 --2- 172.21.15.152:0/25063096 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc7a8060280 0x7fc7a81066e0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:33.405 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.403+0000 7fc7b066f700 1 -- 172.21.15.152:0/25063096 >> 172.21.15.152:0/25063096 conn(0x7fc7a80f0b70 msgr2=0x7fc7a80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:33.405 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.403+0000 7fc7b066f700 1 -- 172.21.15.152:0/25063096 shutdown_connections 2023-12-06T03:56:33.405 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:33.403+0000 7fc7b066f700 1 -- 172.21.15.152:0/25063096 wait complete. 2023-12-06T03:56:33.406 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:33.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:33 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:32.416517+0000 mgr.smithi132.gokxja (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:33.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:33 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:33.402841+0000 mon.smithi132 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.152:0/25063096' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:34.788 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:34.788 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:35.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:35 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:34.416742+0000 mgr.smithi132.gokxja (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:35.955 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.952+0000 7fefb2735700 1 -- 172.21.15.152:0/585833866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac0f51a0 msgr2=0x7fefac0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:35.956 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.952+0000 7fefb2735700 1 --2- 172.21.15.152:0/585833866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac0f51a0 0x7fefac0f55c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fefa8009fb0 tx=0x7fefa8009b30).stop 2023-12-06T03:56:35.956 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.952+0000 7fefb2735700 1 -- 172.21.15.152:0/585833866 shutdown_connections 2023-12-06T03:56:35.956 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.952+0000 7fefb2735700 1 --2- 172.21.15.152:0/585833866 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac0f51a0 0x7fefac0f55c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:35.956 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.952+0000 7fefb2735700 1 -- 172.21.15.152:0/585833866 >> 172.21.15.152:0/585833866 conn(0x7fefac0f09a0 msgr2=0x7fefac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:35.956 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.952+0000 7fefb2735700 1 -- 172.21.15.152:0/585833866 shutdown_connections 2023-12-06T03:56:35.956 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.952+0000 7fefb2735700 1 -- 172.21.15.152:0/585833866 wait complete. 2023-12-06T03:56:35.956 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.952+0000 7fefb2735700 1 Processor -- start 2023-12-06T03:56:35.956 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.952+0000 7fefb2735700 1 -- start start 2023-12-06T03:56:35.957 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.953+0000 7fefb2735700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac0f51a0 0x7fefac1086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:35.957 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.953+0000 7fefb2735700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefac108bc0 con 0x7fefac0f51a0 2023-12-06T03:56:35.957 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.953+0000 7fefb1733700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac0f51a0 0x7fefac1086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:35.957 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.953+0000 7fefb1733700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac0f51a0 0x7fefac1086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40066/0 (socket says 172.21.15.152:40066) 2023-12-06T03:56:35.957 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.953+0000 7fefb1733700 1 -- 172.21.15.152:0/1107325685 learned_addr learned my addr 172.21.15.152:0/1107325685 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:35.957 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.953+0000 7fefb1733700 1 -- 172.21.15.152:0/1107325685 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefa8005040 con 0x7fefac0f51a0 2023-12-06T03:56:35.957 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.953+0000 7fefb1733700 1 --2- 172.21.15.152:0/1107325685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac0f51a0 0x7fefac1086d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fefa8009830 tx=0x7fefa800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:35.957 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.953+0000 7fefa1ffb700 1 -- 172.21.15.152:0/1107325685 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefa8009320 con 0x7fefac0f51a0 2023-12-06T03:56:35.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.954+0000 7fefb2735700 1 -- 172.21.15.152:0/1107325685 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefac108dc0 con 0x7fefac0f51a0 2023-12-06T03:56:35.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.954+0000 7fefa1ffb700 1 -- 172.21.15.152:0/1107325685 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefa80136c0 con 0x7fefac0f51a0 2023-12-06T03:56:35.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.954+0000 7fefa1ffb700 1 -- 172.21.15.152:0/1107325685 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefa801ee90 con 0x7fefac0f51a0 2023-12-06T03:56:35.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.954+0000 7fefb2735700 1 -- 172.21.15.152:0/1107325685 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefac109250 con 0x7fefac0f51a0 2023-12-06T03:56:35.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.954+0000 7fefa1ffb700 1 -- 172.21.15.152:0/1107325685 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefa801e420 con 0x7fefac0f51a0 2023-12-06T03:56:35.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.954+0000 7fefb2735700 1 -- 172.21.15.152:0/1107325685 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefac045ad0 con 0x7fefac0f51a0 2023-12-06T03:56:35.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.955+0000 7fefa1ffb700 1 --2- 172.21.15.152:0/1107325685 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fef9802ff10 0x7fef980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:35.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.955+0000 7fefa1ffb700 1 -- 172.21.15.152:0/1107325685 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefa80410b0 con 0x7fefac0f51a0 2023-12-06T03:56:35.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.955+0000 7fefb0f32700 1 --2- 172.21.15.152:0/1107325685 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fef9802ff10 0x7fef980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:35.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.955+0000 7fefb0f32700 1 --2- 172.21.15.152:0/1107325685 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fef9802ff10 0x7fef980323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fef9c0096a0 tx=0x7fef9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:35.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:35.959+0000 7fefa1ffb700 1 -- 172.21.15.152:0/1107325685 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefa80095f0 con 0x7fefac0f51a0 2023-12-06T03:56:36.168 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:36.167+0000 7fefb2735700 1 -- 172.21.15.152:0/1107325685 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefac10a0a0 con 0x7fefac0f51a0 2023-12-06T03:56:36.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:36.167+0000 7fefa1ffb700 1 -- 172.21.15.152:0/1107325685 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fefa802c2d0 con 0x7fefac0f51a0 2023-12-06T03:56:36.169 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:36.169 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:36.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:36.169+0000 7fefb2735700 1 -- 172.21.15.152:0/1107325685 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fef9802ff10 msgr2=0x7fef980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:36.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:36.169+0000 7fefb2735700 1 --2- 172.21.15.152:0/1107325685 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fef9802ff10 0x7fef980323d0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fef9c0096a0 tx=0x7fef9c006b40).stop 2023-12-06T03:56:36.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:36.169+0000 7fefb2735700 1 -- 172.21.15.152:0/1107325685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac0f51a0 msgr2=0x7fefac1086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:36.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:36.169+0000 7fefb2735700 1 --2- 172.21.15.152:0/1107325685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac0f51a0 0x7fefac1086d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fefa8009830 tx=0x7fefa800b040).stop 2023-12-06T03:56:36.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:36.169+0000 7fefb2735700 1 -- 172.21.15.152:0/1107325685 shutdown_connections 2023-12-06T03:56:36.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:36.169+0000 7fefb2735700 1 --2- 172.21.15.152:0/1107325685 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fef9802ff10 0x7fef980323d0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:36.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:36.169+0000 7fefb2735700 1 --2- 172.21.15.152:0/1107325685 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefac0f51a0 0x7fefac1086d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:36.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:36.169+0000 7fefb2735700 1 -- 172.21.15.152:0/1107325685 >> 172.21.15.152:0/1107325685 conn(0x7fefac0f09a0 msgr2=0x7fefac05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:36.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:36.169+0000 7fefb2735700 1 -- 172.21.15.152:0/1107325685 shutdown_connections 2023-12-06T03:56:36.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:36.169+0000 7fefb2735700 1 -- 172.21.15.152:0/1107325685 wait complete. 2023-12-06T03:56:36.172 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:36.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:36 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:36.168742+0000 mon.smithi132 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.152:0/1107325685' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:37.514 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:37.514 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:37.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:36.417073+0000 mgr.smithi132.gokxja (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:38.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.658+0000 7f35a3a8a700 1 -- 172.21.15.152:0/2716341545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f359c060280 msgr2=0x7f359c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:38.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.658+0000 7f35a3a8a700 1 --2- 172.21.15.152:0/2716341545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f359c060280 0x7f359c0f90f0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f3590009fb0 tx=0x7f3590009b30).stop 2023-12-06T03:56:38.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.658+0000 7f35a3a8a700 1 -- 172.21.15.152:0/2716341545 shutdown_connections 2023-12-06T03:56:38.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.658+0000 7f35a3a8a700 1 --2- 172.21.15.152:0/2716341545 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f359c060280 0x7f359c0f90f0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:38.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.658+0000 7f35a3a8a700 1 -- 172.21.15.152:0/2716341545 >> 172.21.15.152:0/2716341545 conn(0x7f359c0f0b70 msgr2=0x7f359c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:38.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.658+0000 7f35a3a8a700 1 -- 172.21.15.152:0/2716341545 shutdown_connections 2023-12-06T03:56:38.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.658+0000 7f35a3a8a700 1 -- 172.21.15.152:0/2716341545 wait complete. 2023-12-06T03:56:38.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.659+0000 7f35a3a8a700 1 Processor -- start 2023-12-06T03:56:38.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.659+0000 7f35a3a8a700 1 -- start start 2023-12-06T03:56:38.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.659+0000 7f35a3a8a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f359c060280 0x7f359c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:38.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.659+0000 7f35a3a8a700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f359c106bd0 con 0x7f359c060280 2023-12-06T03:56:38.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.659+0000 7f35a1826700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f359c060280 0x7f359c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:38.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.659+0000 7f35a1826700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f359c060280 0x7f359c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40078/0 (socket says 172.21.15.152:40078) 2023-12-06T03:56:38.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.659+0000 7f35a1826700 1 -- 172.21.15.152:0/3844572767 learned_addr learned my addr 172.21.15.152:0/3844572767 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:38.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.659+0000 7f35a1826700 1 -- 172.21.15.152:0/3844572767 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3590005040 con 0x7f359c060280 2023-12-06T03:56:38.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.659+0000 7f35a1826700 1 --2- 172.21.15.152:0/3844572767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f359c060280 0x7f359c1066e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f3590013510 tx=0x7f359000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:38.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.660+0000 7f358e7fc700 1 -- 172.21.15.152:0/3844572767 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3590009320 con 0x7f359c060280 2023-12-06T03:56:38.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.660+0000 7f35a3a8a700 1 -- 172.21.15.152:0/3844572767 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f359c106dd0 con 0x7f359c060280 2023-12-06T03:56:38.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.660+0000 7f358e7fc700 1 -- 172.21.15.152:0/3844572767 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3590013f00 con 0x7f359c060280 2023-12-06T03:56:38.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.660+0000 7f358e7fc700 1 -- 172.21.15.152:0/3844572767 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3590014dc0 con 0x7f359c060280 2023-12-06T03:56:38.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.660+0000 7f35a3a8a700 1 -- 172.21.15.152:0/3844572767 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f359c107260 con 0x7f359c060280 2023-12-06T03:56:38.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.660+0000 7f358e7fc700 1 -- 172.21.15.152:0/3844572767 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3590014420 con 0x7f359c060280 2023-12-06T03:56:38.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.660+0000 7f35a3a8a700 1 -- 172.21.15.152:0/3844572767 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f359c045ad0 con 0x7f359c060280 2023-12-06T03:56:38.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.661+0000 7f358e7fc700 1 --2- 172.21.15.152:0/3844572767 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f358802ff60 0x7f3588032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:38.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.661+0000 7f358e7fc700 1 -- 172.21.15.152:0/3844572767 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3590041050 con 0x7f359c060280 2023-12-06T03:56:38.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.661+0000 7f35a1025700 1 --2- 172.21.15.152:0/3844572767 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f358802ff60 0x7f3588032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:38.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.661+0000 7f35a1025700 1 --2- 172.21.15.152:0/3844572767 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f358802ff60 0x7f3588032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f35980096a0 tx=0x7f3598006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:38.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.665+0000 7f358e7fc700 1 -- 172.21.15.152:0/3844572767 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35900095f0 con 0x7f359c060280 2023-12-06T03:56:38.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.869+0000 7f35a3a8a700 1 -- 172.21.15.152:0/3844572767 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f359c108110 con 0x7f359c060280 2023-12-06T03:56:38.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.870+0000 7f358e7fc700 1 -- 172.21.15.152:0/3844572767 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f359001cb10 con 0x7f359c060280 2023-12-06T03:56:38.872 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:38.872 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:38.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.871+0000 7f35a3a8a700 1 -- 172.21.15.152:0/3844572767 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f358802ff60 msgr2=0x7f3588032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:38.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.871+0000 7f35a3a8a700 1 --2- 172.21.15.152:0/3844572767 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f358802ff60 0x7f3588032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f35980096a0 tx=0x7f3598006b40).stop 2023-12-06T03:56:38.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.871+0000 7f35a3a8a700 1 -- 172.21.15.152:0/3844572767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f359c060280 msgr2=0x7f359c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:38.873 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.871+0000 7f35a3a8a700 1 --2- 172.21.15.152:0/3844572767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f359c060280 0x7f359c1066e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f3590013510 tx=0x7f359000b040).stop 2023-12-06T03:56:38.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.872+0000 7f35a3a8a700 1 -- 172.21.15.152:0/3844572767 shutdown_connections 2023-12-06T03:56:38.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.872+0000 7f35a3a8a700 1 --2- 172.21.15.152:0/3844572767 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f358802ff60 0x7f3588032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:38.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.872+0000 7f35a3a8a700 1 --2- 172.21.15.152:0/3844572767 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f359c060280 0x7f359c1066e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:38.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.872+0000 7f35a3a8a700 1 -- 172.21.15.152:0/3844572767 >> 172.21.15.152:0/3844572767 conn(0x7f359c0f0b70 msgr2=0x7f359c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:38.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.872+0000 7f35a3a8a700 1 -- 172.21.15.152:0/3844572767 shutdown_connections 2023-12-06T03:56:38.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:38.872+0000 7f35a3a8a700 1 -- 172.21.15.152:0/3844572767 wait complete. 2023-12-06T03:56:38.874 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:39.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:39 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:38.417326+0000 mgr.smithi132.gokxja (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:39.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:39 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:38.871502+0000 mon.smithi132 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.152:0/3844572767' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:40.237 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:40.238 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:41.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.336+0000 7fb97097a700 1 -- 172.21.15.152:0/2404401496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb96c0f51a0 msgr2=0x7fb96c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:41.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.336+0000 7fb97097a700 1 --2- 172.21.15.152:0/2404401496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb96c0f51a0 0x7fb96c0f55c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fb95c009fb0 tx=0x7fb95c009b30).stop 2023-12-06T03:56:41.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.336+0000 7fb97097a700 1 -- 172.21.15.152:0/2404401496 shutdown_connections 2023-12-06T03:56:41.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.336+0000 7fb97097a700 1 --2- 172.21.15.152:0/2404401496 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb96c0f51a0 0x7fb96c0f55c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:41.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.336+0000 7fb97097a700 1 -- 172.21.15.152:0/2404401496 >> 172.21.15.152:0/2404401496 conn(0x7fb96c0f09a0 msgr2=0x7fb96c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:41.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.336+0000 7fb97097a700 1 -- 172.21.15.152:0/2404401496 shutdown_connections 2023-12-06T03:56:41.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.336+0000 7fb97097a700 1 -- 172.21.15.152:0/2404401496 wait complete. 2023-12-06T03:56:41.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.337+0000 7fb97097a700 1 Processor -- start 2023-12-06T03:56:41.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.337+0000 7fb97097a700 1 -- start start 2023-12-06T03:56:41.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.337+0000 7fb97097a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb96c0f51a0 0x7fb96c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:41.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.337+0000 7fb97097a700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb96c060220 con 0x7fb96c0f51a0 2023-12-06T03:56:41.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.337+0000 7fb96ad9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb96c0f51a0 0x7fb96c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:41.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.337+0000 7fb96ad9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb96c0f51a0 0x7fb96c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:55238/0 (socket says 172.21.15.152:55238) 2023-12-06T03:56:41.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.337+0000 7fb96ad9d700 1 -- 172.21.15.152:0/1583168031 learned_addr learned my addr 172.21.15.152:0/1583168031 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:41.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.338+0000 7fb96ad9d700 1 -- 172.21.15.152:0/1583168031 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb95c005040 con 0x7fb96c0f51a0 2023-12-06T03:56:41.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.338+0000 7fb96ad9d700 1 --2- 172.21.15.152:0/1583168031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb96c0f51a0 0x7fb96c1091d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fb95c013510 tx=0x7fb95c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:41.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.338+0000 7fb9537fe700 1 -- 172.21.15.152:0/1583168031 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb95c009320 con 0x7fb96c0f51a0 2023-12-06T03:56:41.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.338+0000 7fb97097a700 1 -- 172.21.15.152:0/1583168031 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb96c060420 con 0x7fb96c0f51a0 2023-12-06T03:56:41.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.338+0000 7fb9537fe700 1 -- 172.21.15.152:0/1583168031 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb95c013f00 con 0x7fb96c0f51a0 2023-12-06T03:56:41.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.338+0000 7fb9537fe700 1 -- 172.21.15.152:0/1583168031 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb95c014dc0 con 0x7fb96c0f51a0 2023-12-06T03:56:41.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.338+0000 7fb97097a700 1 -- 172.21.15.152:0/1583168031 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb96c109880 con 0x7fb96c0f51a0 2023-12-06T03:56:41.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.339+0000 7fb9537fe700 1 -- 172.21.15.152:0/1583168031 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb95c014420 con 0x7fb96c0f51a0 2023-12-06T03:56:41.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.339+0000 7fb97097a700 1 -- 172.21.15.152:0/1583168031 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb96c059100 con 0x7fb96c0f51a0 2023-12-06T03:56:41.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.339+0000 7fb9537fe700 1 --2- 172.21.15.152:0/1583168031 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb95402ff60 0x7fb954032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:41.343 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.339+0000 7fb9537fe700 1 -- 172.21.15.152:0/1583168031 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb95c0410f0 con 0x7fb96c0f51a0 2023-12-06T03:56:41.343 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.339+0000 7fb96a59c700 1 --2- 172.21.15.152:0/1583168031 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb95402ff60 0x7fb954032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:41.343 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.340+0000 7fb96a59c700 1 --2- 172.21.15.152:0/1583168031 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb95402ff60 0x7fb954032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fb9600096a0 tx=0x7fb960006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:41.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.344+0000 7fb9537fe700 1 -- 172.21.15.152:0/1583168031 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb95c0095f0 con 0x7fb96c0f51a0 2023-12-06T03:56:41.550 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.548+0000 7fb97097a700 1 -- 172.21.15.152:0/1583168031 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb96c045ad0 con 0x7fb96c0f51a0 2023-12-06T03:56:41.550 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.549+0000 7fb9537fe700 1 -- 172.21.15.152:0/1583168031 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb95c01cb10 con 0x7fb96c0f51a0 2023-12-06T03:56:41.551 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:41.551 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:41.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.550+0000 7fb97097a700 1 -- 172.21.15.152:0/1583168031 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb95402ff60 msgr2=0x7fb954032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:41.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.550+0000 7fb97097a700 1 --2- 172.21.15.152:0/1583168031 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb95402ff60 0x7fb954032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fb9600096a0 tx=0x7fb960006b40).stop 2023-12-06T03:56:41.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.550+0000 7fb97097a700 1 -- 172.21.15.152:0/1583168031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb96c0f51a0 msgr2=0x7fb96c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:41.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.550+0000 7fb97097a700 1 --2- 172.21.15.152:0/1583168031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb96c0f51a0 0x7fb96c1091d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fb95c013510 tx=0x7fb95c00b040).stop 2023-12-06T03:56:41.553 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.550+0000 7fb97097a700 1 -- 172.21.15.152:0/1583168031 shutdown_connections 2023-12-06T03:56:41.553 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.550+0000 7fb97097a700 1 --2- 172.21.15.152:0/1583168031 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb95402ff60 0x7fb954032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:41.553 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.550+0000 7fb97097a700 1 --2- 172.21.15.152:0/1583168031 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb96c0f51a0 0x7fb96c1091d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:41.553 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.550+0000 7fb97097a700 1 -- 172.21.15.152:0/1583168031 >> 172.21.15.152:0/1583168031 conn(0x7fb96c0f09a0 msgr2=0x7fb96c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:41.553 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.550+0000 7fb97097a700 1 -- 172.21.15.152:0/1583168031 shutdown_connections 2023-12-06T03:56:41.553 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:41.550+0000 7fb97097a700 1 -- 172.21.15.152:0/1583168031 wait complete. 2023-12-06T03:56:41.553 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:41.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:41 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:40.417667+0000 mgr.smithi132.gokxja (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:41.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:41 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:41.550297+0000 mon.smithi132 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.152:0/1583168031' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:42.918 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:42.919 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:43.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:42.418009+0000 mgr.smithi132.gokxja (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:44.064 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.061+0000 7f0082f8c700 1 -- 172.21.15.152:0/1195608005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f007c0f51a0 msgr2=0x7f007c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:44.064 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.061+0000 7f0082f8c700 1 --2- 172.21.15.152:0/1195608005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f007c0f51a0 0x7f007c0f55c0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f0078009fb0 tx=0x7f0078009b30).stop 2023-12-06T03:56:44.064 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.062+0000 7f0082f8c700 1 -- 172.21.15.152:0/1195608005 shutdown_connections 2023-12-06T03:56:44.064 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.062+0000 7f0082f8c700 1 --2- 172.21.15.152:0/1195608005 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f007c0f51a0 0x7f007c0f55c0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:44.064 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.062+0000 7f0082f8c700 1 -- 172.21.15.152:0/1195608005 >> 172.21.15.152:0/1195608005 conn(0x7f007c0f09a0 msgr2=0x7f007c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:44.064 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.062+0000 7f0082f8c700 1 -- 172.21.15.152:0/1195608005 shutdown_connections 2023-12-06T03:56:44.064 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.062+0000 7f0082f8c700 1 -- 172.21.15.152:0/1195608005 wait complete. 2023-12-06T03:56:44.064 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.062+0000 7f0082f8c700 1 Processor -- start 2023-12-06T03:56:44.064 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.062+0000 7f0082f8c700 1 -- start start 2023-12-06T03:56:44.066 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.063+0000 7f0082f8c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f007c0f51a0 0x7f007c108f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:44.067 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.063+0000 7f0082f8c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f007c060240 con 0x7f007c0f51a0 2023-12-06T03:56:44.067 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.063+0000 7f0081f8a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f007c0f51a0 0x7f007c108f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:44.067 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.063+0000 7f0081f8a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f007c0f51a0 0x7f007c108f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:55250/0 (socket says 172.21.15.152:55250) 2023-12-06T03:56:44.071 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.063+0000 7f0081f8a700 1 -- 172.21.15.152:0/3835814961 learned_addr learned my addr 172.21.15.152:0/3835814961 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:44.071 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.063+0000 7f0081f8a700 1 -- 172.21.15.152:0/3835814961 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0078005040 con 0x7f007c0f51a0 2023-12-06T03:56:44.071 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.063+0000 7f0081f8a700 1 --2- 172.21.15.152:0/3835814961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f007c0f51a0 0x7f007c108f90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f0078002040 tx=0x7f007800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:44.072 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.063+0000 7f00727fc700 1 -- 172.21.15.152:0/3835814961 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0078008ad0 con 0x7f007c0f51a0 2023-12-06T03:56:44.072 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.063+0000 7f0082f8c700 1 -- 172.21.15.152:0/3835814961 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f007c060440 con 0x7f007c0f51a0 2023-12-06T03:56:44.072 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.063+0000 7f00727fc700 1 -- 172.21.15.152:0/3835814961 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0078008c30 con 0x7f007c0f51a0 2023-12-06T03:56:44.072 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.063+0000 7f00727fc700 1 -- 172.21.15.152:0/3835814961 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0078014420 con 0x7f007c0f51a0 2023-12-06T03:56:44.072 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.063+0000 7f0082f8c700 1 -- 172.21.15.152:0/3835814961 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f007c109640 con 0x7f007c0f51a0 2023-12-06T03:56:44.072 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.064+0000 7f00727fc700 1 -- 172.21.15.152:0/3835814961 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0078008da0 con 0x7f007c0f51a0 2023-12-06T03:56:44.072 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.064+0000 7f0082f8c700 1 -- 172.21.15.152:0/3835814961 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f007c045ad0 con 0x7f007c0f51a0 2023-12-06T03:56:44.074 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.064+0000 7f00727fc700 1 --2- 172.21.15.152:0/3835814961 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f006802ff60 0x7f0068032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:44.074 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.064+0000 7f00727fc700 1 -- 172.21.15.152:0/3835814961 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f00780412a0 con 0x7f007c0f51a0 2023-12-06T03:56:44.074 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.065+0000 7f0081789700 1 --2- 172.21.15.152:0/3835814961 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f006802ff60 0x7f0068032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:44.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.065+0000 7f0081789700 1 --2- 172.21.15.152:0/3835814961 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f006802ff60 0x7f0068032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f006c0096a0 tx=0x7f006c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:44.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.069+0000 7f00727fc700 1 -- 172.21.15.152:0/3835814961 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f007801e420 con 0x7f007c0f51a0 2023-12-06T03:56:44.276 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.275+0000 7f0082f8c700 1 -- 172.21.15.152:0/3835814961 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f007c0fabe0 con 0x7f007c0f51a0 2023-12-06T03:56:44.277 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.275+0000 7f00727fc700 1 -- 172.21.15.152:0/3835814961 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f007802c2d0 con 0x7f007c0f51a0 2023-12-06T03:56:44.277 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:44.277 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:44.279 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.277+0000 7f0082f8c700 1 -- 172.21.15.152:0/3835814961 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f006802ff60 msgr2=0x7f0068032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:44.279 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.277+0000 7f0082f8c700 1 --2- 172.21.15.152:0/3835814961 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f006802ff60 0x7f0068032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f006c0096a0 tx=0x7f006c006b40).stop 2023-12-06T03:56:44.279 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.277+0000 7f0082f8c700 1 -- 172.21.15.152:0/3835814961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f007c0f51a0 msgr2=0x7f007c108f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:44.279 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.277+0000 7f0082f8c700 1 --2- 172.21.15.152:0/3835814961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f007c0f51a0 0x7f007c108f90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f0078002040 tx=0x7f007800b040).stop 2023-12-06T03:56:44.279 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.277+0000 7f0082f8c700 1 -- 172.21.15.152:0/3835814961 shutdown_connections 2023-12-06T03:56:44.279 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.277+0000 7f0082f8c700 1 --2- 172.21.15.152:0/3835814961 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f006802ff60 0x7f0068032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:44.279 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.277+0000 7f0082f8c700 1 --2- 172.21.15.152:0/3835814961 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f007c0f51a0 0x7f007c108f90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:44.279 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.277+0000 7f0082f8c700 1 -- 172.21.15.152:0/3835814961 >> 172.21.15.152:0/3835814961 conn(0x7f007c0f09a0 msgr2=0x7f007c0f11b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:44.279 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.277+0000 7f0082f8c700 1 -- 172.21.15.152:0/3835814961 shutdown_connections 2023-12-06T03:56:44.279 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:44.277+0000 7f0082f8c700 1 -- 172.21.15.152:0/3835814961 wait complete. 2023-12-06T03:56:44.280 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:45.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:44 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:44.276722+0000 mon.smithi132 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.152:0/3835814961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:45.649 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:45.649 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:46.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:45 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:44.418318+0000 mgr.smithi132.gokxja (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:46.772 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.770+0000 7facf923e700 1 -- 172.21.15.152:0/1979832160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf40f51a0 msgr2=0x7facf40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:46.772 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.770+0000 7facf923e700 1 --2- 172.21.15.152:0/1979832160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf40f51a0 0x7facf40f55c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7face4009fb0 tx=0x7face4009b30).stop 2023-12-06T03:56:46.772 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.770+0000 7facf923e700 1 -- 172.21.15.152:0/1979832160 shutdown_connections 2023-12-06T03:56:46.772 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.770+0000 7facf923e700 1 --2- 172.21.15.152:0/1979832160 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf40f51a0 0x7facf40f55c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:46.772 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.770+0000 7facf923e700 1 -- 172.21.15.152:0/1979832160 >> 172.21.15.152:0/1979832160 conn(0x7facf40f09a0 msgr2=0x7facf40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:46.772 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.770+0000 7facf923e700 1 -- 172.21.15.152:0/1979832160 shutdown_connections 2023-12-06T03:56:46.772 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.770+0000 7facf923e700 1 -- 172.21.15.152:0/1979832160 wait complete. 2023-12-06T03:56:46.772 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.770+0000 7facf923e700 1 Processor -- start 2023-12-06T03:56:46.774 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.771+0000 7facf923e700 1 -- start start 2023-12-06T03:56:46.774 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.771+0000 7facf923e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf40f51a0 0x7facf40fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:46.774 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.771+0000 7facf923e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7facf40fb5a0 con 0x7facf40f51a0 2023-12-06T03:56:46.774 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.771+0000 7facf3fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf40f51a0 0x7facf40fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:46.774 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.771+0000 7facf3fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf40f51a0 0x7facf40fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:55270/0 (socket says 172.21.15.152:55270) 2023-12-06T03:56:46.774 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.771+0000 7facf3fff700 1 -- 172.21.15.152:0/4041720165 learned_addr learned my addr 172.21.15.152:0/4041720165 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:46.775 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.771+0000 7facf3fff700 1 -- 172.21.15.152:0/4041720165 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face4005040 con 0x7facf40f51a0 2023-12-06T03:56:46.775 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.771+0000 7facf3fff700 1 --2- 172.21.15.152:0/4041720165 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf40f51a0 0x7facf40fd120 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7face4013510 tx=0x7face400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:46.775 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.771+0000 7facf0ff9700 1 -- 172.21.15.152:0/4041720165 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7face4009320 con 0x7facf40f51a0 2023-12-06T03:56:46.775 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.771+0000 7facf0ff9700 1 -- 172.21.15.152:0/4041720165 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7face4013f00 con 0x7facf40f51a0 2023-12-06T03:56:46.775 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.771+0000 7facf923e700 1 -- 172.21.15.152:0/4041720165 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facf40fb740 con 0x7facf40f51a0 2023-12-06T03:56:46.775 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.771+0000 7facf0ff9700 1 -- 172.21.15.152:0/4041720165 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7face4014dc0 con 0x7facf40f51a0 2023-12-06T03:56:46.775 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.772+0000 7facf923e700 1 -- 172.21.15.152:0/4041720165 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7facf40fbbd0 con 0x7facf40f51a0 2023-12-06T03:56:46.775 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.772+0000 7facf0ff9700 1 -- 172.21.15.152:0/4041720165 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7face40095f0 con 0x7facf40f51a0 2023-12-06T03:56:46.775 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.772+0000 7facf923e700 1 -- 172.21.15.152:0/4041720165 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facf4059100 con 0x7facf40f51a0 2023-12-06T03:56:46.775 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.772+0000 7facf0ff9700 1 --2- 172.21.15.152:0/4041720165 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7facdc02ff60 0x7facdc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:46.775 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.772+0000 7facf0ff9700 1 -- 172.21.15.152:0/4041720165 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7face40410c0 con 0x7facf40f51a0 2023-12-06T03:56:46.775 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.773+0000 7facf37fe700 1 --2- 172.21.15.152:0/4041720165 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7facdc02ff60 0x7facdc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:46.776 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.773+0000 7facf37fe700 1 --2- 172.21.15.152:0/4041720165 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7facdc02ff60 0x7facdc032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7face80096a0 tx=0x7face8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:46.779 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.777+0000 7facf0ff9700 1 -- 172.21.15.152:0/4041720165 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7face401c790 con 0x7facf40f51a0 2023-12-06T03:56:46.984 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.982+0000 7facf923e700 1 -- 172.21.15.152:0/4041720165 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7facf40fabe0 con 0x7facf40f51a0 2023-12-06T03:56:46.985 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.983+0000 7facf0ff9700 1 -- 172.21.15.152:0/4041720165 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7face4010230 con 0x7facf40f51a0 2023-12-06T03:56:46.986 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:46.986 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:46.987 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.984+0000 7facf923e700 1 -- 172.21.15.152:0/4041720165 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7facdc02ff60 msgr2=0x7facdc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:46.987 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.984+0000 7facf923e700 1 --2- 172.21.15.152:0/4041720165 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7facdc02ff60 0x7facdc032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7face80096a0 tx=0x7face8006b40).stop 2023-12-06T03:56:46.987 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.984+0000 7facf923e700 1 -- 172.21.15.152:0/4041720165 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf40f51a0 msgr2=0x7facf40fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:46.987 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.984+0000 7facf923e700 1 --2- 172.21.15.152:0/4041720165 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf40f51a0 0x7facf40fd120 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7face4013510 tx=0x7face400b040).stop 2023-12-06T03:56:46.988 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.984+0000 7facf923e700 1 -- 172.21.15.152:0/4041720165 shutdown_connections 2023-12-06T03:56:46.988 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.984+0000 7facf923e700 1 --2- 172.21.15.152:0/4041720165 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7facdc02ff60 0x7facdc032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:46.988 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.984+0000 7facf923e700 1 --2- 172.21.15.152:0/4041720165 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7facf40f51a0 0x7facf40fd120 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:46.988 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.984+0000 7facf923e700 1 -- 172.21.15.152:0/4041720165 >> 172.21.15.152:0/4041720165 conn(0x7facf40f09a0 msgr2=0x7facf405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:46.988 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.984+0000 7facf923e700 1 -- 172.21.15.152:0/4041720165 shutdown_connections 2023-12-06T03:56:46.988 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:46.984+0000 7facf923e700 1 -- 172.21.15.152:0/4041720165 wait complete. 2023-12-06T03:56:46.988 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:48.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:47 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:46.418644+0000 mgr.smithi132.gokxja (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:48.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:47 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:46.983924+0000 mon.smithi132 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.152:0/4041720165' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:48.358 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:48.359 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:49.495 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.490+0000 7fcd23ad5700 1 -- 172.21.15.152:0/554286854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd1c0f5410 msgr2=0x7fcd1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:49.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.490+0000 7fcd23ad5700 1 --2- 172.21.15.152:0/554286854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd1c0f5410 0x7fcd1c0f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fcd0c009fb0 tx=0x7fcd0c009b30).stop 2023-12-06T03:56:49.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.490+0000 7fcd23ad5700 1 -- 172.21.15.152:0/554286854 shutdown_connections 2023-12-06T03:56:49.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.490+0000 7fcd23ad5700 1 --2- 172.21.15.152:0/554286854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd1c0f5410 0x7fcd1c0f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:49.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.490+0000 7fcd23ad5700 1 -- 172.21.15.152:0/554286854 >> 172.21.15.152:0/554286854 conn(0x7fcd1c0f0b70 msgr2=0x7fcd1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:49.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.491+0000 7fcd23ad5700 1 -- 172.21.15.152:0/554286854 shutdown_connections 2023-12-06T03:56:49.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.491+0000 7fcd23ad5700 1 -- 172.21.15.152:0/554286854 wait complete. 2023-12-06T03:56:49.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.491+0000 7fcd23ad5700 1 Processor -- start 2023-12-06T03:56:49.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.491+0000 7fcd23ad5700 1 -- start start 2023-12-06T03:56:49.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.491+0000 7fcd23ad5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd1c0f5410 0x7fcd1c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:49.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.491+0000 7fcd23ad5700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd1c105b20 con 0x7fcd1c0f5410 2023-12-06T03:56:49.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.492+0000 7fcd21871700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd1c0f5410 0x7fcd1c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:49.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.492+0000 7fcd21871700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd1c0f5410 0x7fcd1c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:42204/0 (socket says 172.21.15.152:42204) 2023-12-06T03:56:49.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.492+0000 7fcd21871700 1 -- 172.21.15.152:0/749661039 learned_addr learned my addr 172.21.15.152:0/749661039 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:49.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.492+0000 7fcd21871700 1 -- 172.21.15.152:0/749661039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd0c005040 con 0x7fcd1c0f5410 2023-12-06T03:56:49.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.492+0000 7fcd21871700 1 --2- 172.21.15.152:0/749661039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd1c0f5410 0x7fcd1c105630 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fcd0c013510 tx=0x7fcd0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:49.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.492+0000 7fcd127fc700 1 -- 172.21.15.152:0/749661039 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd0c009320 con 0x7fcd1c0f5410 2023-12-06T03:56:49.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.492+0000 7fcd23ad5700 1 -- 172.21.15.152:0/749661039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd1c105d20 con 0x7fcd1c0f5410 2023-12-06T03:56:49.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.492+0000 7fcd127fc700 1 -- 172.21.15.152:0/749661039 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd0c013f00 con 0x7fcd1c0f5410 2023-12-06T03:56:49.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.492+0000 7fcd127fc700 1 -- 172.21.15.152:0/749661039 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd0c014d90 con 0x7fcd1c0f5410 2023-12-06T03:56:49.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.492+0000 7fcd23ad5700 1 -- 172.21.15.152:0/749661039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd1c1061b0 con 0x7fcd1c0f5410 2023-12-06T03:56:49.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.493+0000 7fcd127fc700 1 -- 172.21.15.152:0/749661039 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd0c0095f0 con 0x7fcd1c0f5410 2023-12-06T03:56:49.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.493+0000 7fcd23ad5700 1 -- 172.21.15.152:0/749661039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd1c045ad0 con 0x7fcd1c0f5410 2023-12-06T03:56:49.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.493+0000 7fcd127fc700 1 --2- 172.21.15.152:0/749661039 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd0802ff60 0x7fcd08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:49.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.493+0000 7fcd127fc700 1 -- 172.21.15.152:0/749661039 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd0c041090 con 0x7fcd1c0f5410 2023-12-06T03:56:49.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.493+0000 7fcd21070700 1 --2- 172.21.15.152:0/749661039 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd0802ff60 0x7fcd08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:49.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.494+0000 7fcd21070700 1 --2- 172.21.15.152:0/749661039 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd0802ff60 0x7fcd08032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fcd180096a0 tx=0x7fcd18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:49.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.498+0000 7fcd127fc700 1 -- 172.21.15.152:0/749661039 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd0c014420 con 0x7fcd1c0f5410 2023-12-06T03:56:49.709 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.707+0000 7fcd23ad5700 1 -- 172.21.15.152:0/749661039 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd1c0fae60 con 0x7fcd1c0f5410 2023-12-06T03:56:49.710 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.708+0000 7fcd127fc700 1 -- 172.21.15.152:0/749661039 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcd0c01d7f0 con 0x7fcd1c0f5410 2023-12-06T03:56:49.710 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:49.711 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:49.712 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.710+0000 7fcd23ad5700 1 -- 172.21.15.152:0/749661039 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd0802ff60 msgr2=0x7fcd08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:49.712 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.710+0000 7fcd23ad5700 1 --2- 172.21.15.152:0/749661039 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd0802ff60 0x7fcd08032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fcd180096a0 tx=0x7fcd18006b40).stop 2023-12-06T03:56:49.712 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.710+0000 7fcd23ad5700 1 -- 172.21.15.152:0/749661039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd1c0f5410 msgr2=0x7fcd1c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:49.712 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.710+0000 7fcd23ad5700 1 --2- 172.21.15.152:0/749661039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd1c0f5410 0x7fcd1c105630 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fcd0c013510 tx=0x7fcd0c00b040).stop 2023-12-06T03:56:49.712 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.710+0000 7fcd23ad5700 1 -- 172.21.15.152:0/749661039 shutdown_connections 2023-12-06T03:56:49.713 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.710+0000 7fcd23ad5700 1 --2- 172.21.15.152:0/749661039 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcd0802ff60 0x7fcd08032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:49.713 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.710+0000 7fcd23ad5700 1 --2- 172.21.15.152:0/749661039 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcd1c0f5410 0x7fcd1c105630 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:49.713 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.710+0000 7fcd23ad5700 1 -- 172.21.15.152:0/749661039 >> 172.21.15.152:0/749661039 conn(0x7fcd1c0f0b70 msgr2=0x7fcd1c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:49.713 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.710+0000 7fcd23ad5700 1 -- 172.21.15.152:0/749661039 shutdown_connections 2023-12-06T03:56:49.713 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:49.710+0000 7fcd23ad5700 1 -- 172.21.15.152:0/749661039 wait complete. 2023-12-06T03:56:49.713 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:50.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:49 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:48.418893+0000 mgr.smithi132.gokxja (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:51.090 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:51.090 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:51.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:50 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:49.709530+0000 mon.smithi132 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.152:0/749661039' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:52.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:51 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:50.419240+0000 mgr.smithi132.gokxja (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:52.221 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.218+0000 7fbc6b284700 1 -- 172.21.15.152:0/3810771433 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc640f5410 msgr2=0x7fbc640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:52.221 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.218+0000 7fbc6b284700 1 --2- 172.21.15.152:0/3810771433 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc640f5410 0x7fbc640f5830 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fbc60009fb0 tx=0x7fbc60009b30).stop 2023-12-06T03:56:52.221 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.218+0000 7fbc6b284700 1 -- 172.21.15.152:0/3810771433 shutdown_connections 2023-12-06T03:56:52.221 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.218+0000 7fbc6b284700 1 --2- 172.21.15.152:0/3810771433 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc640f5410 0x7fbc640f5830 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:52.221 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.218+0000 7fbc6b284700 1 -- 172.21.15.152:0/3810771433 >> 172.21.15.152:0/3810771433 conn(0x7fbc640f0b70 msgr2=0x7fbc640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:52.221 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.218+0000 7fbc6b284700 1 -- 172.21.15.152:0/3810771433 shutdown_connections 2023-12-06T03:56:52.221 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.218+0000 7fbc6b284700 1 -- 172.21.15.152:0/3810771433 wait complete. 2023-12-06T03:56:52.221 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.219+0000 7fbc6b284700 1 Processor -- start 2023-12-06T03:56:52.221 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.219+0000 7fbc6b284700 1 -- start start 2023-12-06T03:56:52.221 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.219+0000 7fbc6b284700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc640f5410 0x7fbc64109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:52.221 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.219+0000 7fbc6b284700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc64060220 con 0x7fbc640f5410 2023-12-06T03:56:52.222 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.219+0000 7fbc69020700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc640f5410 0x7fbc64109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:52.222 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.219+0000 7fbc69020700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc640f5410 0x7fbc64109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:42208/0 (socket says 172.21.15.152:42208) 2023-12-06T03:56:52.222 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.219+0000 7fbc69020700 1 -- 172.21.15.152:0/2907495991 learned_addr learned my addr 172.21.15.152:0/2907495991 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:52.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.219+0000 7fbc69020700 1 -- 172.21.15.152:0/2907495991 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc60005040 con 0x7fbc640f5410 2023-12-06T03:56:52.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.220+0000 7fbc69020700 1 --2- 172.21.15.152:0/2907495991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc640f5410 0x7fbc64109450 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fbc60013510 tx=0x7fbc6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:52.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.220+0000 7fbc59ffb700 1 -- 172.21.15.152:0/2907495991 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc60009320 con 0x7fbc640f5410 2023-12-06T03:56:52.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.220+0000 7fbc6b284700 1 -- 172.21.15.152:0/2907495991 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc64060420 con 0x7fbc640f5410 2023-12-06T03:56:52.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.220+0000 7fbc59ffb700 1 -- 172.21.15.152:0/2907495991 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc60013f00 con 0x7fbc640f5410 2023-12-06T03:56:52.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.220+0000 7fbc59ffb700 1 -- 172.21.15.152:0/2907495991 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc60014d90 con 0x7fbc640f5410 2023-12-06T03:56:52.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.220+0000 7fbc6b284700 1 -- 172.21.15.152:0/2907495991 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc64109b00 con 0x7fbc640f5410 2023-12-06T03:56:52.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.221+0000 7fbc59ffb700 1 -- 172.21.15.152:0/2907495991 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc600095f0 con 0x7fbc640f5410 2023-12-06T03:56:52.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.221+0000 7fbc6b284700 1 -- 172.21.15.152:0/2907495991 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc64059100 con 0x7fbc640f5410 2023-12-06T03:56:52.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.221+0000 7fbc59ffb700 1 --2- 172.21.15.152:0/2907495991 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbc5002ffb0 0x7fbc50032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:52.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.221+0000 7fbc59ffb700 1 -- 172.21.15.152:0/2907495991 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc60040fe0 con 0x7fbc640f5410 2023-12-06T03:56:52.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.221+0000 7fbc6881f700 1 --2- 172.21.15.152:0/2907495991 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbc5002ffb0 0x7fbc50032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:52.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.221+0000 7fbc6881f700 1 --2- 172.21.15.152:0/2907495991 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbc5002ffb0 0x7fbc50032470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fbc540096a0 tx=0x7fbc54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:52.227 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.226+0000 7fbc59ffb700 1 -- 172.21.15.152:0/2907495991 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc6006d2b0 con 0x7fbc640f5410 2023-12-06T03:56:52.435 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.434+0000 7fbc6b284700 1 -- 172.21.15.152:0/2907495991 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc64045ad0 con 0x7fbc640f5410 2023-12-06T03:56:52.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.435+0000 7fbc59ffb700 1 -- 172.21.15.152:0/2907495991 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbc60014420 con 0x7fbc640f5410 2023-12-06T03:56:52.436 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:52.436 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:52.438 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.436+0000 7fbc6b284700 1 -- 172.21.15.152:0/2907495991 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbc5002ffb0 msgr2=0x7fbc50032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:52.438 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.436+0000 7fbc6b284700 1 --2- 172.21.15.152:0/2907495991 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbc5002ffb0 0x7fbc50032470 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fbc540096a0 tx=0x7fbc54006b40).stop 2023-12-06T03:56:52.438 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.436+0000 7fbc6b284700 1 -- 172.21.15.152:0/2907495991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc640f5410 msgr2=0x7fbc64109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:52.438 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.436+0000 7fbc6b284700 1 --2- 172.21.15.152:0/2907495991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc640f5410 0x7fbc64109450 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fbc60013510 tx=0x7fbc6000b040).stop 2023-12-06T03:56:52.438 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.436+0000 7fbc6b284700 1 -- 172.21.15.152:0/2907495991 shutdown_connections 2023-12-06T03:56:52.438 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.436+0000 7fbc6b284700 1 --2- 172.21.15.152:0/2907495991 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbc5002ffb0 0x7fbc50032470 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:52.438 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.436+0000 7fbc6b284700 1 --2- 172.21.15.152:0/2907495991 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc640f5410 0x7fbc64109450 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:52.438 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.436+0000 7fbc6b284700 1 -- 172.21.15.152:0/2907495991 >> 172.21.15.152:0/2907495991 conn(0x7fbc640f0b70 msgr2=0x7fbc640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:52.438 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.436+0000 7fbc6b284700 1 -- 172.21.15.152:0/2907495991 shutdown_connections 2023-12-06T03:56:52.439 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:52.436+0000 7fbc6b284700 1 -- 172.21.15.152:0/2907495991 wait complete. 2023-12-06T03:56:52.439 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:53.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:52 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:52.435926+0000 mon.smithi132 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.152:0/2907495991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:53.809 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:53.810 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:54.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:53 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:52.419619+0000 mgr.smithi132.gokxja (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:54.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.886+0000 7f265f59e700 1 -- 172.21.15.152:0/3888580486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26600f51a0 msgr2=0x7f26600f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:54.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.886+0000 7f265f59e700 1 --2- 172.21.15.152:0/3888580486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26600f51a0 0x7f26600f55c0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f2648009fb0 tx=0x7f2648009b30).stop 2023-12-06T03:56:54.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.886+0000 7f265f59e700 1 -- 172.21.15.152:0/3888580486 shutdown_connections 2023-12-06T03:56:54.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.886+0000 7f265f59e700 1 --2- 172.21.15.152:0/3888580486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26600f51a0 0x7f26600f55c0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.886+0000 7f265f59e700 1 -- 172.21.15.152:0/3888580486 >> 172.21.15.152:0/3888580486 conn(0x7f26600f09a0 msgr2=0x7f26600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.886+0000 7f265f59e700 1 -- 172.21.15.152:0/3888580486 shutdown_connections 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.886+0000 7f265f59e700 1 -- 172.21.15.152:0/3888580486 wait complete. 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.887+0000 7f265f59e700 1 Processor -- start 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.887+0000 7f265f59e700 1 -- start start 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.887+0000 7f265f59e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26600f51a0 0x7f26601064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.887+0000 7f265f59e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26601069d0 con 0x7f26600f51a0 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.887+0000 7f265e59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26600f51a0 0x7f26601064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.887+0000 7f265e59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26600f51a0 0x7f26601064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:42224/0 (socket says 172.21.15.152:42224) 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.887+0000 7f265e59c700 1 -- 172.21.15.152:0/4258304616 learned_addr learned my addr 172.21.15.152:0/4258304616 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.887+0000 7f265e59c700 1 -- 172.21.15.152:0/4258304616 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2648005040 con 0x7f26600f51a0 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.888+0000 7f265e59c700 1 --2- 172.21.15.152:0/4258304616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26600f51a0 0x7f26601064e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f2648013510 tx=0x7f264800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.888+0000 7f2656ffd700 1 -- 172.21.15.152:0/4258304616 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2648009320 con 0x7f26600f51a0 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.888+0000 7f265f59e700 1 -- 172.21.15.152:0/4258304616 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2660106bd0 con 0x7f26600f51a0 2023-12-06T03:56:54.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.888+0000 7f2656ffd700 1 -- 172.21.15.152:0/4258304616 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2648013f00 con 0x7f26600f51a0 2023-12-06T03:56:54.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.888+0000 7f2656ffd700 1 -- 172.21.15.152:0/4258304616 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2648014dc0 con 0x7f26600f51a0 2023-12-06T03:56:54.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.888+0000 7f265f59e700 1 -- 172.21.15.152:0/4258304616 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2660107060 con 0x7f26600f51a0 2023-12-06T03:56:54.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.888+0000 7f2656ffd700 1 -- 172.21.15.152:0/4258304616 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26480095f0 con 0x7f26600f51a0 2023-12-06T03:56:54.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.889+0000 7f265f59e700 1 -- 172.21.15.152:0/4258304616 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2660045ad0 con 0x7f26600f51a0 2023-12-06T03:56:54.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.889+0000 7f2656ffd700 1 --2- 172.21.15.152:0/4258304616 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f264c02ffb0 0x7f264c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:54.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.889+0000 7f2656ffd700 1 -- 172.21.15.152:0/4258304616 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2648040ff0 con 0x7f26600f51a0 2023-12-06T03:56:54.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.889+0000 7f265dd9b700 1 --2- 172.21.15.152:0/4258304616 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f264c02ffb0 0x7f264c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:54.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.889+0000 7f265dd9b700 1 --2- 172.21.15.152:0/4258304616 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f264c02ffb0 0x7f264c032470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f26500096a0 tx=0x7f2650006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:54.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:54.893+0000 7f2656ffd700 1 -- 172.21.15.152:0/4258304616 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f264806d2b0 con 0x7f26600f51a0 2023-12-06T03:56:55.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:55.100+0000 7f265f59e700 1 -- 172.21.15.152:0/4258304616 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2660107f10 con 0x7f26600f51a0 2023-12-06T03:56:55.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:55.101+0000 7f2656ffd700 1 -- 172.21.15.152:0/4258304616 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2648014420 con 0x7f26600f51a0 2023-12-06T03:56:55.103 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:55.103 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:55.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:55.102+0000 7f265f59e700 1 -- 172.21.15.152:0/4258304616 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f264c02ffb0 msgr2=0x7f264c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:55.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:55.102+0000 7f265f59e700 1 --2- 172.21.15.152:0/4258304616 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f264c02ffb0 0x7f264c032470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f26500096a0 tx=0x7f2650006b40).stop 2023-12-06T03:56:55.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:55.102+0000 7f265f59e700 1 -- 172.21.15.152:0/4258304616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26600f51a0 msgr2=0x7f26601064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:55.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:55.102+0000 7f265f59e700 1 --2- 172.21.15.152:0/4258304616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26600f51a0 0x7f26601064e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f2648013510 tx=0x7f264800b040).stop 2023-12-06T03:56:55.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:55.102+0000 7f265f59e700 1 -- 172.21.15.152:0/4258304616 shutdown_connections 2023-12-06T03:56:55.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:55.103+0000 7f265f59e700 1 --2- 172.21.15.152:0/4258304616 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f264c02ffb0 0x7f264c032470 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:55.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:55.103+0000 7f265f59e700 1 --2- 172.21.15.152:0/4258304616 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26600f51a0 0x7f26601064e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:55.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:55.103+0000 7f265f59e700 1 -- 172.21.15.152:0/4258304616 >> 172.21.15.152:0/4258304616 conn(0x7f26600f09a0 msgr2=0x7f266005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:55.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:55.103+0000 7f265f59e700 1 -- 172.21.15.152:0/4258304616 shutdown_connections 2023-12-06T03:56:55.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:55.103+0000 7f265f59e700 1 -- 172.21.15.152:0/4258304616 wait complete. 2023-12-06T03:56:55.106 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:55.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:54.519269+0000 mon.smithi132 (mon.0) 200 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/mirror_snapshot_schedule"}]: dispatch 2023-12-06T03:56:55.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:54.527970+0000 mon.smithi132 (mon.0) 201 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/trash_purge_schedule"}]: dispatch 2023-12-06T03:56:56.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:54.419884+0000 mgr.smithi132.gokxja (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:56.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:55.102383+0000 mon.smithi132 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.152:0/4258304616' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:56.467 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:56.467 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:56:57.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.542+0000 7feaa6452700 1 -- 172.21.15.152:0/409842736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feaa00f5410 msgr2=0x7feaa00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:57.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.542+0000 7feaa6452700 1 --2- 172.21.15.152:0/409842736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feaa00f5410 0x7feaa00f5830 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7fea88009fb0 tx=0x7fea88009b30).stop 2023-12-06T03:56:57.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.542+0000 7feaa6452700 1 -- 172.21.15.152:0/409842736 shutdown_connections 2023-12-06T03:56:57.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.542+0000 7feaa6452700 1 --2- 172.21.15.152:0/409842736 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feaa00f5410 0x7feaa00f5830 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:57.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.542+0000 7feaa6452700 1 -- 172.21.15.152:0/409842736 >> 172.21.15.152:0/409842736 conn(0x7feaa00f0b70 msgr2=0x7feaa00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:57.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.542+0000 7feaa6452700 1 -- 172.21.15.152:0/409842736 shutdown_connections 2023-12-06T03:56:57.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.542+0000 7feaa6452700 1 -- 172.21.15.152:0/409842736 wait complete. 2023-12-06T03:56:57.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.543+0000 7feaa6452700 1 Processor -- start 2023-12-06T03:56:57.545 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.543+0000 7feaa6452700 1 -- start start 2023-12-06T03:56:57.547 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.543+0000 7feaa6452700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feaa00f5410 0x7feaa0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:57.547 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.543+0000 7feaa6452700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaa0105b20 con 0x7feaa00f5410 2023-12-06T03:56:57.547 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.543+0000 7fea9ffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feaa00f5410 0x7feaa0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:57.547 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.544+0000 7fea9ffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feaa00f5410 0x7feaa0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:42248/0 (socket says 172.21.15.152:42248) 2023-12-06T03:56:57.547 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.544+0000 7fea9ffff700 1 -- 172.21.15.152:0/3879530857 learned_addr learned my addr 172.21.15.152:0/3879530857 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:56:57.547 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.544+0000 7fea9ffff700 1 -- 172.21.15.152:0/3879530857 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea88005040 con 0x7feaa00f5410 2023-12-06T03:56:57.547 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.544+0000 7fea9ffff700 1 --2- 172.21.15.152:0/3879530857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feaa00f5410 0x7feaa0105630 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fea88013510 tx=0x7fea8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:57.547 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.544+0000 7fea9cff9700 1 -- 172.21.15.152:0/3879530857 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea88009320 con 0x7feaa00f5410 2023-12-06T03:56:57.548 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.544+0000 7feaa6452700 1 -- 172.21.15.152:0/3879530857 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaa0105d20 con 0x7feaa00f5410 2023-12-06T03:56:57.548 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.544+0000 7fea9cff9700 1 -- 172.21.15.152:0/3879530857 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea88013f00 con 0x7feaa00f5410 2023-12-06T03:56:57.548 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.544+0000 7fea9cff9700 1 -- 172.21.15.152:0/3879530857 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea88014d90 con 0x7feaa00f5410 2023-12-06T03:56:57.548 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.544+0000 7feaa6452700 1 -- 172.21.15.152:0/3879530857 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaa01061b0 con 0x7feaa00f5410 2023-12-06T03:56:57.548 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.545+0000 7fea9cff9700 1 -- 172.21.15.152:0/3879530857 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea88014420 con 0x7feaa00f5410 2023-12-06T03:56:57.548 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.545+0000 7feaa6452700 1 -- 172.21.15.152:0/3879530857 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaa0045ad0 con 0x7feaa00f5410 2023-12-06T03:56:57.548 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.545+0000 7fea9cff9700 1 --2- 172.21.15.152:0/3879530857 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fea8c02ff60 0x7fea8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:56:57.548 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.545+0000 7fea9cff9700 1 -- 172.21.15.152:0/3879530857 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea880410c0 con 0x7feaa00f5410 2023-12-06T03:56:57.548 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.545+0000 7fea9f7fe700 1 --2- 172.21.15.152:0/3879530857 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fea8c02ff60 0x7fea8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:56:57.548 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.546+0000 7fea9f7fe700 1 --2- 172.21.15.152:0/3879530857 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fea8c02ff60 0x7fea8c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fea900096a0 tx=0x7fea90006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:56:57.553 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.550+0000 7fea9cff9700 1 -- 172.21.15.152:0/3879530857 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea880095f0 con 0x7feaa00f5410 2023-12-06T03:56:57.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.754+0000 7feaa6452700 1 -- 172.21.15.152:0/3879530857 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feaa00fae60 con 0x7feaa00f5410 2023-12-06T03:56:57.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.755+0000 7fea9cff9700 1 -- 172.21.15.152:0/3879530857 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fea8801db70 con 0x7feaa00f5410 2023-12-06T03:56:57.757 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:56:57.757 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:56:57.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.757+0000 7feaa6452700 1 -- 172.21.15.152:0/3879530857 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fea8c02ff60 msgr2=0x7fea8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:57.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.757+0000 7feaa6452700 1 --2- 172.21.15.152:0/3879530857 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fea8c02ff60 0x7fea8c032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fea900096a0 tx=0x7fea90006b40).stop 2023-12-06T03:56:57.759 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.757+0000 7feaa6452700 1 -- 172.21.15.152:0/3879530857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feaa00f5410 msgr2=0x7feaa0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:56:57.759 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.757+0000 7feaa6452700 1 --2- 172.21.15.152:0/3879530857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feaa00f5410 0x7feaa0105630 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7fea88013510 tx=0x7fea8800b040).stop 2023-12-06T03:56:57.759 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.757+0000 7feaa6452700 1 -- 172.21.15.152:0/3879530857 shutdown_connections 2023-12-06T03:56:57.759 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.757+0000 7feaa6452700 1 --2- 172.21.15.152:0/3879530857 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fea8c02ff60 0x7fea8c032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:57.759 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.757+0000 7feaa6452700 1 --2- 172.21.15.152:0/3879530857 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feaa00f5410 0x7feaa0105630 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:56:57.759 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.757+0000 7feaa6452700 1 -- 172.21.15.152:0/3879530857 >> 172.21.15.152:0/3879530857 conn(0x7feaa00f0b70 msgr2=0x7feaa00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:56:57.759 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.757+0000 7feaa6452700 1 -- 172.21.15.152:0/3879530857 shutdown_connections 2023-12-06T03:56:57.759 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:56:57.757+0000 7feaa6452700 1 -- 172.21.15.152:0/3879530857 wait complete. 2023-12-06T03:56:57.759 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:56:58.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:56.420240+0000 mgr.smithi132.gokxja (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:56:59.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:58 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:56:57.756399+0000 mon.smithi132 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.152:0/3879530857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:56:59.128 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:56:59.128 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:00.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:56:59 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:56:58.420582+0000 mgr.smithi132.gokxja (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:00.246 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.244+0000 7f1052a25700 1 -- 172.21.15.152:0/462222617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f104c060280 msgr2=0x7f104c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:00.247 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.244+0000 7f1052a25700 1 --2- 172.21.15.152:0/462222617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f104c060280 0x7f104c0f90f0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f103c009fb0 tx=0x7f103c009b30).stop 2023-12-06T03:57:00.247 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.244+0000 7f1052a25700 1 -- 172.21.15.152:0/462222617 shutdown_connections 2023-12-06T03:57:00.247 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.244+0000 7f1052a25700 1 --2- 172.21.15.152:0/462222617 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f104c060280 0x7f104c0f90f0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:00.247 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.244+0000 7f1052a25700 1 -- 172.21.15.152:0/462222617 >> 172.21.15.152:0/462222617 conn(0x7f104c0f0b70 msgr2=0x7f104c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:00.247 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.244+0000 7f1052a25700 1 -- 172.21.15.152:0/462222617 shutdown_connections 2023-12-06T03:57:00.247 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.244+0000 7f1052a25700 1 -- 172.21.15.152:0/462222617 wait complete. 2023-12-06T03:57:00.247 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.244+0000 7f1052a25700 1 Processor -- start 2023-12-06T03:57:00.247 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.245+0000 7f1052a25700 1 -- start start 2023-12-06T03:57:00.247 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.245+0000 7f1052a25700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f104c060280 0x7f104c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:00.247 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.245+0000 7f1052a25700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f104c106bd0 con 0x7f104c060280 2023-12-06T03:57:00.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.245+0000 7f104bfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f104c060280 0x7f104c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:00.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.245+0000 7f104bfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f104c060280 0x7f104c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:55952/0 (socket says 172.21.15.152:55952) 2023-12-06T03:57:00.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.245+0000 7f104bfff700 1 -- 172.21.15.152:0/4150883609 learned_addr learned my addr 172.21.15.152:0/4150883609 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:00.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.245+0000 7f104bfff700 1 -- 172.21.15.152:0/4150883609 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f103c005040 con 0x7f104c060280 2023-12-06T03:57:00.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.246+0000 7f104bfff700 1 --2- 172.21.15.152:0/4150883609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f104c060280 0x7f104c1066e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f103c013510 tx=0x7f103c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:00.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.246+0000 7f1048ff9700 1 -- 172.21.15.152:0/4150883609 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f103c009320 con 0x7f104c060280 2023-12-06T03:57:00.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.246+0000 7f1052a25700 1 -- 172.21.15.152:0/4150883609 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f104c106dd0 con 0x7f104c060280 2023-12-06T03:57:00.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.246+0000 7f1048ff9700 1 -- 172.21.15.152:0/4150883609 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f103c013f00 con 0x7f104c060280 2023-12-06T03:57:00.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.246+0000 7f1048ff9700 1 -- 172.21.15.152:0/4150883609 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f103c014dc0 con 0x7f104c060280 2023-12-06T03:57:00.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.246+0000 7f1052a25700 1 -- 172.21.15.152:0/4150883609 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f104c107260 con 0x7f104c060280 2023-12-06T03:57:00.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.246+0000 7f1048ff9700 1 -- 172.21.15.152:0/4150883609 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f103c01d3d0 con 0x7f104c060280 2023-12-06T03:57:00.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.247+0000 7f1048ff9700 1 --2- 172.21.15.152:0/4150883609 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f103402ff60 0x7f1034032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:00.250 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.247+0000 7f1048ff9700 1 -- 172.21.15.152:0/4150883609 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f103c0413c0 con 0x7f104c060280 2023-12-06T03:57:00.250 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.247+0000 7f1052a25700 1 -- 172.21.15.152:0/4150883609 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f104c059100 con 0x7f104c060280 2023-12-06T03:57:00.250 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.247+0000 7f104b7fe700 1 --2- 172.21.15.152:0/4150883609 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f103402ff60 0x7f1034032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:00.250 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.247+0000 7f104b7fe700 1 --2- 172.21.15.152:0/4150883609 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f103402ff60 0x7f1034032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f10400096a0 tx=0x7f1040006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:00.253 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.252+0000 7f1048ff9700 1 -- 172.21.15.152:0/4150883609 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f103c012920 con 0x7f104c060280 2023-12-06T03:57:00.463 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.461+0000 7f1052a25700 1 -- 172.21.15.152:0/4150883609 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f104c045ad0 con 0x7f104c060280 2023-12-06T03:57:00.463 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.462+0000 7f1048ff9700 1 -- 172.21.15.152:0/4150883609 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f103c02b030 con 0x7f104c060280 2023-12-06T03:57:00.464 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:00.464 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:00.465 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.463+0000 7f1052a25700 1 -- 172.21.15.152:0/4150883609 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f103402ff60 msgr2=0x7f1034032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:00.465 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.463+0000 7f1052a25700 1 --2- 172.21.15.152:0/4150883609 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f103402ff60 0x7f1034032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f10400096a0 tx=0x7f1040006b40).stop 2023-12-06T03:57:00.465 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.463+0000 7f1052a25700 1 -- 172.21.15.152:0/4150883609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f104c060280 msgr2=0x7f104c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:00.465 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.463+0000 7f1052a25700 1 --2- 172.21.15.152:0/4150883609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f104c060280 0x7f104c1066e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f103c013510 tx=0x7f103c00b040).stop 2023-12-06T03:57:00.465 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.463+0000 7f1052a25700 1 -- 172.21.15.152:0/4150883609 shutdown_connections 2023-12-06T03:57:00.465 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.463+0000 7f1052a25700 1 --2- 172.21.15.152:0/4150883609 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f103402ff60 0x7f1034032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:00.465 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.463+0000 7f1052a25700 1 --2- 172.21.15.152:0/4150883609 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f104c060280 0x7f104c1066e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:00.465 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.463+0000 7f1052a25700 1 -- 172.21.15.152:0/4150883609 >> 172.21.15.152:0/4150883609 conn(0x7f104c0f0b70 msgr2=0x7f104c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:00.465 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.463+0000 7f1052a25700 1 -- 172.21.15.152:0/4150883609 shutdown_connections 2023-12-06T03:57:00.465 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:00.463+0000 7f1052a25700 1 -- 172.21.15.152:0/4150883609 wait complete. 2023-12-06T03:57:00.466 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:01.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:00 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:00.463086+0000 mon.smithi132 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.152:0/4150883609' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:01.847 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:01.848 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:02.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:01 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:00.420998+0000 mgr.smithi132.gokxja (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:02.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.959+0000 7fbb92e2d700 1 -- 172.21.15.152:0/431643924 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c0f6510 msgr2=0x7fbb8c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:02.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.959+0000 7fbb92e2d700 1 --2- 172.21.15.152:0/431643924 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c0f6510 0x7fbb8c0f6930 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fbb80009fb0 tx=0x7fbb80009b30).stop 2023-12-06T03:57:02.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.959+0000 7fbb92e2d700 1 -- 172.21.15.152:0/431643924 shutdown_connections 2023-12-06T03:57:02.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.959+0000 7fbb92e2d700 1 --2- 172.21.15.152:0/431643924 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c0f6510 0x7fbb8c0f6930 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:02.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.959+0000 7fbb92e2d700 1 -- 172.21.15.152:0/431643924 >> 172.21.15.152:0/431643924 conn(0x7fbb8c0f1c70 msgr2=0x7fbb8c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:02.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.959+0000 7fbb92e2d700 1 -- 172.21.15.152:0/431643924 shutdown_connections 2023-12-06T03:57:02.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.959+0000 7fbb92e2d700 1 -- 172.21.15.152:0/431643924 wait complete. 2023-12-06T03:57:02.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.960+0000 7fbb92e2d700 1 Processor -- start 2023-12-06T03:57:02.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.960+0000 7fbb92e2d700 1 -- start start 2023-12-06T03:57:02.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.960+0000 7fbb92e2d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c0f6510 0x7fbb8c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:02.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.960+0000 7fbb92e2d700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb8c106c20 con 0x7fbb8c0f6510 2023-12-06T03:57:02.964 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.960+0000 7fbb90bc9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c0f6510 0x7fbb8c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:02.964 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.960+0000 7fbb90bc9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c0f6510 0x7fbb8c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:55970/0 (socket says 172.21.15.152:55970) 2023-12-06T03:57:02.964 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.960+0000 7fbb90bc9700 1 -- 172.21.15.152:0/1099389467 learned_addr learned my addr 172.21.15.152:0/1099389467 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:02.964 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.961+0000 7fbb90bc9700 1 -- 172.21.15.152:0/1099389467 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb80005040 con 0x7fbb8c0f6510 2023-12-06T03:57:02.964 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.961+0000 7fbb90bc9700 1 --2- 172.21.15.152:0/1099389467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c0f6510 0x7fbb8c106730 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fbb80013510 tx=0x7fbb8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:02.964 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.961+0000 7fbb897fa700 1 -- 172.21.15.152:0/1099389467 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb80009320 con 0x7fbb8c0f6510 2023-12-06T03:57:02.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.961+0000 7fbb897fa700 1 -- 172.21.15.152:0/1099389467 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb80013f00 con 0x7fbb8c0f6510 2023-12-06T03:57:02.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.961+0000 7fbb92e2d700 1 -- 172.21.15.152:0/1099389467 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb8c106e20 con 0x7fbb8c0f6510 2023-12-06T03:57:02.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.961+0000 7fbb897fa700 1 -- 172.21.15.152:0/1099389467 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb80014d90 con 0x7fbb8c0f6510 2023-12-06T03:57:02.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.961+0000 7fbb92e2d700 1 -- 172.21.15.152:0/1099389467 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb8c1072b0 con 0x7fbb8c0f6510 2023-12-06T03:57:02.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.962+0000 7fbb897fa700 1 -- 172.21.15.152:0/1099389467 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb800263a0 con 0x7fbb8c0f6510 2023-12-06T03:57:02.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.962+0000 7fbb897fa700 1 --2- 172.21.15.152:0/1099389467 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb7c02ff60 0x7fbb7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:02.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.962+0000 7fbb92e2d700 1 -- 172.21.15.152:0/1099389467 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb8c045ad0 con 0x7fbb8c0f6510 2023-12-06T03:57:02.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.962+0000 7fbb897fa700 1 -- 172.21.15.152:0/1099389467 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb800412f0 con 0x7fbb8c0f6510 2023-12-06T03:57:02.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.962+0000 7fbb8bfff700 1 --2- 172.21.15.152:0/1099389467 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb7c02ff60 0x7fbb7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:02.965 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.962+0000 7fbb8bfff700 1 --2- 172.21.15.152:0/1099389467 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb7c02ff60 0x7fbb7c032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fbb780096a0 tx=0x7fbb78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:02.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:02.967+0000 7fbb897fa700 1 -- 172.21.15.152:0/1099389467 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb80012920 con 0x7fbb8c0f6510 2023-12-06T03:57:03.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:03.171+0000 7fbb92e2d700 1 -- 172.21.15.152:0/1099389467 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb8c0fc050 con 0x7fbb8c0f6510 2023-12-06T03:57:03.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:03.172+0000 7fbb897fa700 1 -- 172.21.15.152:0/1099389467 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbb8002b030 con 0x7fbb8c0f6510 2023-12-06T03:57:03.174 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:03.174 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:03.175 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:03.174+0000 7fbb92e2d700 1 -- 172.21.15.152:0/1099389467 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb7c02ff60 msgr2=0x7fbb7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:03.175 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:03.174+0000 7fbb92e2d700 1 --2- 172.21.15.152:0/1099389467 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb7c02ff60 0x7fbb7c032420 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7fbb780096a0 tx=0x7fbb78006b40).stop 2023-12-06T03:57:03.176 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:03.174+0000 7fbb92e2d700 1 -- 172.21.15.152:0/1099389467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c0f6510 msgr2=0x7fbb8c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:03.176 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:03.174+0000 7fbb92e2d700 1 --2- 172.21.15.152:0/1099389467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c0f6510 0x7fbb8c106730 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fbb80013510 tx=0x7fbb8000b040).stop 2023-12-06T03:57:03.176 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:03.174+0000 7fbb92e2d700 1 -- 172.21.15.152:0/1099389467 shutdown_connections 2023-12-06T03:57:03.176 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:03.174+0000 7fbb92e2d700 1 --2- 172.21.15.152:0/1099389467 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbb7c02ff60 0x7fbb7c032420 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:03.176 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:03.174+0000 7fbb92e2d700 1 --2- 172.21.15.152:0/1099389467 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbb8c0f6510 0x7fbb8c106730 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:03.176 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:03.174+0000 7fbb92e2d700 1 -- 172.21.15.152:0/1099389467 >> 172.21.15.152:0/1099389467 conn(0x7fbb8c0f1c70 msgr2=0x7fbb8c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:03.176 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:03.174+0000 7fbb92e2d700 1 -- 172.21.15.152:0/1099389467 shutdown_connections 2023-12-06T03:57:03.176 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:03.174+0000 7fbb92e2d700 1 -- 172.21.15.152:0/1099389467 wait complete. 2023-12-06T03:57:03.176 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:04.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:02.421418+0000 mgr.smithi132.gokxja (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:04.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:03.173313+0000 mon.smithi132 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.152:0/1099389467' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:04.556 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:04.557 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:05.692 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.685+0000 7fc2f127f700 1 -- 172.21.15.152:0/3927514494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec060280 msgr2=0x7fc2ec0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:05.692 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.685+0000 7fc2f127f700 1 --2- 172.21.15.152:0/3927514494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec060280 0x7fc2ec0f90f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7fc2e0009fb0 tx=0x7fc2e0009b30).stop 2023-12-06T03:57:05.692 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.685+0000 7fc2f127f700 1 -- 172.21.15.152:0/3927514494 shutdown_connections 2023-12-06T03:57:05.692 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.685+0000 7fc2f127f700 1 --2- 172.21.15.152:0/3927514494 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec060280 0x7fc2ec0f90f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:05.692 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.685+0000 7fc2f127f700 1 -- 172.21.15.152:0/3927514494 >> 172.21.15.152:0/3927514494 conn(0x7fc2ec0f0b70 msgr2=0x7fc2ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:05.692 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.685+0000 7fc2f127f700 1 -- 172.21.15.152:0/3927514494 shutdown_connections 2023-12-06T03:57:05.692 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.685+0000 7fc2f127f700 1 -- 172.21.15.152:0/3927514494 wait complete. 2023-12-06T03:57:05.692 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.685+0000 7fc2f127f700 1 Processor -- start 2023-12-06T03:57:05.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.685+0000 7fc2f127f700 1 -- start start 2023-12-06T03:57:05.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.686+0000 7fc2f127f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec060280 0x7fc2ec1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:05.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.686+0000 7fc2f127f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2ec106bd0 con 0x7fc2ec060280 2023-12-06T03:57:05.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.686+0000 7fc2eaffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec060280 0x7fc2ec1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:05.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.686+0000 7fc2eaffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec060280 0x7fc2ec1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:55996/0 (socket says 172.21.15.152:55996) 2023-12-06T03:57:05.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.686+0000 7fc2eaffd700 1 -- 172.21.15.152:0/1234476134 learned_addr learned my addr 172.21.15.152:0/1234476134 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:05.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.686+0000 7fc2eaffd700 1 -- 172.21.15.152:0/1234476134 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2e0005040 con 0x7fc2ec060280 2023-12-06T03:57:05.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.686+0000 7fc2eaffd700 1 --2- 172.21.15.152:0/1234476134 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec060280 0x7fc2ec1066e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fc2e0013510 tx=0x7fc2e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:05.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.686+0000 7fc2d37fe700 1 -- 172.21.15.152:0/1234476134 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e0009320 con 0x7fc2ec060280 2023-12-06T03:57:05.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.686+0000 7fc2d37fe700 1 -- 172.21.15.152:0/1234476134 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc2e0013f00 con 0x7fc2ec060280 2023-12-06T03:57:05.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.686+0000 7fc2f127f700 1 -- 172.21.15.152:0/1234476134 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2ec106dd0 con 0x7fc2ec060280 2023-12-06T03:57:05.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.686+0000 7fc2d37fe700 1 -- 172.21.15.152:0/1234476134 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc2e0014dc0 con 0x7fc2ec060280 2023-12-06T03:57:05.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.686+0000 7fc2f127f700 1 -- 172.21.15.152:0/1234476134 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2ec107260 con 0x7fc2ec060280 2023-12-06T03:57:05.694 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.687+0000 7fc2d37fe700 1 -- 172.21.15.152:0/1234476134 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc2e00095f0 con 0x7fc2ec060280 2023-12-06T03:57:05.694 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.687+0000 7fc2f127f700 1 -- 172.21.15.152:0/1234476134 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2ec045ad0 con 0x7fc2ec060280 2023-12-06T03:57:05.694 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.687+0000 7fc2d37fe700 1 --2- 172.21.15.152:0/1234476134 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc2d402ff60 0x7fc2d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:05.694 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.688+0000 7fc2d37fe700 1 -- 172.21.15.152:0/1234476134 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc2e0041040 con 0x7fc2ec060280 2023-12-06T03:57:05.694 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.688+0000 7fc2ea7fc700 1 --2- 172.21.15.152:0/1234476134 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc2d402ff60 0x7fc2d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:05.694 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.688+0000 7fc2ea7fc700 1 --2- 172.21.15.152:0/1234476134 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc2d402ff60 0x7fc2d4032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fc2dc0096a0 tx=0x7fc2dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:05.695 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.692+0000 7fc2d37fe700 1 -- 172.21.15.152:0/1234476134 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc2e0014420 con 0x7fc2ec060280 2023-12-06T03:57:05.901 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.899+0000 7fc2f127f700 1 -- 172.21.15.152:0/1234476134 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2ec108110 con 0x7fc2ec060280 2023-12-06T03:57:05.901 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.899+0000 7fc2d37fe700 1 -- 172.21.15.152:0/1234476134 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc2e0010230 con 0x7fc2ec060280 2023-12-06T03:57:05.902 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:05.902 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:05.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.901+0000 7fc2f127f700 1 -- 172.21.15.152:0/1234476134 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc2d402ff60 msgr2=0x7fc2d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:05.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.901+0000 7fc2f127f700 1 --2- 172.21.15.152:0/1234476134 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc2d402ff60 0x7fc2d4032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fc2dc0096a0 tx=0x7fc2dc006b40).stop 2023-12-06T03:57:05.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.901+0000 7fc2f127f700 1 -- 172.21.15.152:0/1234476134 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec060280 msgr2=0x7fc2ec1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:05.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.901+0000 7fc2f127f700 1 --2- 172.21.15.152:0/1234476134 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec060280 0x7fc2ec1066e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fc2e0013510 tx=0x7fc2e000b040).stop 2023-12-06T03:57:05.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.901+0000 7fc2f127f700 1 -- 172.21.15.152:0/1234476134 shutdown_connections 2023-12-06T03:57:05.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.901+0000 7fc2f127f700 1 --2- 172.21.15.152:0/1234476134 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc2d402ff60 0x7fc2d4032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:05.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.901+0000 7fc2f127f700 1 --2- 172.21.15.152:0/1234476134 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc2ec060280 0x7fc2ec1066e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:05.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.901+0000 7fc2f127f700 1 -- 172.21.15.152:0/1234476134 >> 172.21.15.152:0/1234476134 conn(0x7fc2ec0f0b70 msgr2=0x7fc2ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:05.903 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.901+0000 7fc2f127f700 1 -- 172.21.15.152:0/1234476134 shutdown_connections 2023-12-06T03:57:05.904 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:05.901+0000 7fc2f127f700 1 -- 172.21.15.152:0/1234476134 wait complete. 2023-12-06T03:57:05.904 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:06.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:04.421826+0000 mgr.smithi132.gokxja (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:07.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:06 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:05.900883+0000 mon.smithi132 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.152:0/1234476134' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:07.295 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:07.296 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:08.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:07 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:06.422204+0000 mgr.smithi132.gokxja (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:08.423 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.419+0000 7f38bcb75700 1 -- 172.21.15.152:0/654617068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38b80f6510 msgr2=0x7f38b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:08.423 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.419+0000 7f38bcb75700 1 --2- 172.21.15.152:0/654617068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38b80f6510 0x7f38b80f6930 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f38a8009fb0 tx=0x7f38a8009b30).stop 2023-12-06T03:57:08.423 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.419+0000 7f38bcb75700 1 -- 172.21.15.152:0/654617068 shutdown_connections 2023-12-06T03:57:08.423 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.419+0000 7f38bcb75700 1 --2- 172.21.15.152:0/654617068 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38b80f6510 0x7f38b80f6930 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:08.423 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.419+0000 7f38bcb75700 1 -- 172.21.15.152:0/654617068 >> 172.21.15.152:0/654617068 conn(0x7f38b80f1c70 msgr2=0x7f38b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:08.423 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.419+0000 7f38bcb75700 1 -- 172.21.15.152:0/654617068 shutdown_connections 2023-12-06T03:57:08.423 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.419+0000 7f38bcb75700 1 -- 172.21.15.152:0/654617068 wait complete. 2023-12-06T03:57:08.423 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.420+0000 7f38bcb75700 1 Processor -- start 2023-12-06T03:57:08.423 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.420+0000 7f38bcb75700 1 -- start start 2023-12-06T03:57:08.423 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.420+0000 7f38bcb75700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38b80f6510 0x7f38b8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:08.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.420+0000 7f38bcb75700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38b8106c20 con 0x7f38b80f6510 2023-12-06T03:57:08.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.420+0000 7f38b659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38b80f6510 0x7f38b8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:08.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.420+0000 7f38b659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38b80f6510 0x7f38b8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:56028/0 (socket says 172.21.15.152:56028) 2023-12-06T03:57:08.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.420+0000 7f38b659c700 1 -- 172.21.15.152:0/1675166263 learned_addr learned my addr 172.21.15.152:0/1675166263 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:08.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.420+0000 7f38b659c700 1 -- 172.21.15.152:0/1675166263 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38a8005040 con 0x7f38b80f6510 2023-12-06T03:57:08.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.421+0000 7f38b659c700 1 --2- 172.21.15.152:0/1675166263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38b80f6510 0x7f38b8106730 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f38a8014000 tx=0x7f38a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:08.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.421+0000 7f38aeffd700 1 -- 172.21.15.152:0/1675166263 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38a80095f0 con 0x7f38b80f6510 2023-12-06T03:57:08.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.421+0000 7f38bcb75700 1 -- 172.21.15.152:0/1675166263 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38b8106e20 con 0x7f38b80f6510 2023-12-06T03:57:08.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.421+0000 7f38aeffd700 1 -- 172.21.15.152:0/1675166263 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38a8007e70 con 0x7f38b80f6510 2023-12-06T03:57:08.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.421+0000 7f38aeffd700 1 -- 172.21.15.152:0/1675166263 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38a8014ea0 con 0x7f38b80f6510 2023-12-06T03:57:08.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.421+0000 7f38bcb75700 1 -- 172.21.15.152:0/1675166263 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38b81072b0 con 0x7f38b80f6510 2023-12-06T03:57:08.425 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.421+0000 7f38aeffd700 1 -- 172.21.15.152:0/1675166263 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38a8025070 con 0x7f38b80f6510 2023-12-06T03:57:08.425 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.422+0000 7f38aeffd700 1 --2- 172.21.15.152:0/1675166263 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f38a402ff60 0x7f38a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:08.425 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.422+0000 7f38aeffd700 1 -- 172.21.15.152:0/1675166263 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38a8040a30 con 0x7f38b80f6510 2023-12-06T03:57:08.425 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.422+0000 7f38bcb75700 1 -- 172.21.15.152:0/1675166263 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38b8059100 con 0x7f38b80f6510 2023-12-06T03:57:08.425 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.422+0000 7f38b5d9b700 1 --2- 172.21.15.152:0/1675166263 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f38a402ff60 0x7f38a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:08.425 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.422+0000 7f38b5d9b700 1 --2- 172.21.15.152:0/1675166263 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f38a402ff60 0x7f38a4032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f38a00096a0 tx=0x7f38a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:08.428 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.427+0000 7f38aeffd700 1 -- 172.21.15.152:0/1675166263 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38a8026350 con 0x7f38b80f6510 2023-12-06T03:57:08.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.634+0000 7f38bcb75700 1 -- 172.21.15.152:0/1675166263 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38b80fbf60 con 0x7f38b80f6510 2023-12-06T03:57:08.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.635+0000 7f38aeffd700 1 -- 172.21.15.152:0/1675166263 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f38a8025300 con 0x7f38b80f6510 2023-12-06T03:57:08.636 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:08.637 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:08.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.636+0000 7f38bcb75700 1 -- 172.21.15.152:0/1675166263 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f38a402ff60 msgr2=0x7f38a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:08.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.636+0000 7f38bcb75700 1 --2- 172.21.15.152:0/1675166263 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f38a402ff60 0x7f38a4032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f38a00096a0 tx=0x7f38a0006b40).stop 2023-12-06T03:57:08.639 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.636+0000 7f38bcb75700 1 -- 172.21.15.152:0/1675166263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38b80f6510 msgr2=0x7f38b8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:08.639 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.636+0000 7f38bcb75700 1 --2- 172.21.15.152:0/1675166263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38b80f6510 0x7f38b8106730 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f38a8014000 tx=0x7f38a800b040).stop 2023-12-06T03:57:08.639 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.636+0000 7f38bcb75700 1 -- 172.21.15.152:0/1675166263 shutdown_connections 2023-12-06T03:57:08.639 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.636+0000 7f38bcb75700 1 --2- 172.21.15.152:0/1675166263 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f38a402ff60 0x7f38a4032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:08.639 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.636+0000 7f38bcb75700 1 --2- 172.21.15.152:0/1675166263 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f38b80f6510 0x7f38b8106730 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:08.639 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.636+0000 7f38bcb75700 1 -- 172.21.15.152:0/1675166263 >> 172.21.15.152:0/1675166263 conn(0x7f38b80f1c70 msgr2=0x7f38b80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:08.639 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.636+0000 7f38bcb75700 1 -- 172.21.15.152:0/1675166263 shutdown_connections 2023-12-06T03:57:08.639 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:08.636+0000 7f38bcb75700 1 -- 172.21.15.152:0/1675166263 wait complete. 2023-12-06T03:57:08.639 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:09.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:08 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:08.636100+0000 mon.smithi132 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.152:0/1675166263' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:10.043 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:10.043 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:10.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:09 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:08.422536+0000 mgr.smithi132.gokxja (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:11.130 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.127+0000 7f528fb53700 1 -- 172.21.15.152:0/1644561657 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52880f5410 msgr2=0x7f52880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:11.130 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.127+0000 7f528fb53700 1 --2- 172.21.15.152:0/1644561657 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52880f5410 0x7f52880f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f5278009fb0 tx=0x7f5278009b30).stop 2023-12-06T03:57:11.131 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.127+0000 7f528fb53700 1 -- 172.21.15.152:0/1644561657 shutdown_connections 2023-12-06T03:57:11.131 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.127+0000 7f528fb53700 1 --2- 172.21.15.152:0/1644561657 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52880f5410 0x7f52880f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:11.131 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.127+0000 7f528fb53700 1 -- 172.21.15.152:0/1644561657 >> 172.21.15.152:0/1644561657 conn(0x7f52880f0b70 msgr2=0x7f52880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:11.131 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.127+0000 7f528fb53700 1 -- 172.21.15.152:0/1644561657 shutdown_connections 2023-12-06T03:57:11.131 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.127+0000 7f528fb53700 1 -- 172.21.15.152:0/1644561657 wait complete. 2023-12-06T03:57:11.131 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.127+0000 7f528fb53700 1 Processor -- start 2023-12-06T03:57:11.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.127+0000 7f528fb53700 1 -- start start 2023-12-06T03:57:11.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.128+0000 7f528fb53700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52880f5410 0x7f5288105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:11.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.128+0000 7f528fb53700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5288105b20 con 0x7f52880f5410 2023-12-06T03:57:11.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.128+0000 7f528d8ef700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52880f5410 0x7f5288105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:11.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.128+0000 7f528d8ef700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52880f5410 0x7f5288105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:47776/0 (socket says 172.21.15.152:47776) 2023-12-06T03:57:11.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.128+0000 7f528d8ef700 1 -- 172.21.15.152:0/420632092 learned_addr learned my addr 172.21.15.152:0/420632092 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:11.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.128+0000 7f528d8ef700 1 -- 172.21.15.152:0/420632092 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5278005040 con 0x7f52880f5410 2023-12-06T03:57:11.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.128+0000 7f528d8ef700 1 --2- 172.21.15.152:0/420632092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52880f5410 0x7f5288105630 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f5278013510 tx=0x7f527800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:11.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.128+0000 7f527e7fc700 1 -- 172.21.15.152:0/420632092 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5278009320 con 0x7f52880f5410 2023-12-06T03:57:11.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.128+0000 7f528fb53700 1 -- 172.21.15.152:0/420632092 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5288105d20 con 0x7f52880f5410 2023-12-06T03:57:11.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.128+0000 7f527e7fc700 1 -- 172.21.15.152:0/420632092 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5278013f00 con 0x7f52880f5410 2023-12-06T03:57:11.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.128+0000 7f527e7fc700 1 -- 172.21.15.152:0/420632092 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5278014d90 con 0x7f52880f5410 2023-12-06T03:57:11.132 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.128+0000 7f528fb53700 1 -- 172.21.15.152:0/420632092 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52881061b0 con 0x7f52880f5410 2023-12-06T03:57:11.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.129+0000 7f528fb53700 1 -- 172.21.15.152:0/420632092 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5288045ad0 con 0x7f52880f5410 2023-12-06T03:57:11.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.129+0000 7f527e7fc700 1 -- 172.21.15.152:0/420632092 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f52780095f0 con 0x7f52880f5410 2023-12-06T03:57:11.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.129+0000 7f527e7fc700 1 --2- 172.21.15.152:0/420632092 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f527402ff60 0x7f5274032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:11.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.129+0000 7f527e7fc700 1 -- 172.21.15.152:0/420632092 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5278040fc0 con 0x7f52880f5410 2023-12-06T03:57:11.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.130+0000 7f528d0ee700 1 --2- 172.21.15.152:0/420632092 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f527402ff60 0x7f5274032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:11.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.130+0000 7f528d0ee700 1 --2- 172.21.15.152:0/420632092 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f527402ff60 0x7f5274032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f52840096a0 tx=0x7f5284006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:11.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.134+0000 7f527e7fc700 1 -- 172.21.15.152:0/420632092 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5278014420 con 0x7f52880f5410 2023-12-06T03:57:11.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.340+0000 7f528fb53700 1 -- 172.21.15.152:0/420632092 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52880fae60 con 0x7f52880f5410 2023-12-06T03:57:11.343 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.341+0000 7f527e7fc700 1 -- 172.21.15.152:0/420632092 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5278010230 con 0x7f52880f5410 2023-12-06T03:57:11.343 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:11.343 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:11.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.343+0000 7f528fb53700 1 -- 172.21.15.152:0/420632092 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f527402ff60 msgr2=0x7f5274032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:11.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.343+0000 7f528fb53700 1 --2- 172.21.15.152:0/420632092 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f527402ff60 0x7f5274032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f52840096a0 tx=0x7f5284006b40).stop 2023-12-06T03:57:11.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.343+0000 7f528fb53700 1 -- 172.21.15.152:0/420632092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52880f5410 msgr2=0x7f5288105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:11.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.343+0000 7f528fb53700 1 --2- 172.21.15.152:0/420632092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52880f5410 0x7f5288105630 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f5278013510 tx=0x7f527800b040).stop 2023-12-06T03:57:11.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.343+0000 7f528fb53700 1 -- 172.21.15.152:0/420632092 shutdown_connections 2023-12-06T03:57:11.346 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.343+0000 7f528fb53700 1 --2- 172.21.15.152:0/420632092 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f527402ff60 0x7f5274032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:11.346 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.343+0000 7f528fb53700 1 --2- 172.21.15.152:0/420632092 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f52880f5410 0x7f5288105630 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:11.346 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.343+0000 7f528fb53700 1 -- 172.21.15.152:0/420632092 >> 172.21.15.152:0/420632092 conn(0x7f52880f0b70 msgr2=0x7f52880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:11.346 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.343+0000 7f528fb53700 1 -- 172.21.15.152:0/420632092 shutdown_connections 2023-12-06T03:57:11.346 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:11.343+0000 7f528fb53700 1 -- 172.21.15.152:0/420632092 wait complete. 2023-12-06T03:57:11.346 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:12.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:11 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:10.422906+0000 mgr.smithi132.gokxja (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:12.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:11 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:11.342577+0000 mon.smithi132 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.152:0/420632092' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:12.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:12.735 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:13.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.890+0000 7fdb232c4700 1 -- 172.21.15.152:0/3662711916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb1c0f5410 msgr2=0x7fdb1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:13.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.890+0000 7fdb232c4700 1 --2- 172.21.15.152:0/3662711916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb1c0f5410 0x7fdb1c0f5830 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fdb0c009fb0 tx=0x7fdb0c009b30).stop 2023-12-06T03:57:13.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.890+0000 7fdb232c4700 1 -- 172.21.15.152:0/3662711916 shutdown_connections 2023-12-06T03:57:13.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.890+0000 7fdb232c4700 1 --2- 172.21.15.152:0/3662711916 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb1c0f5410 0x7fdb1c0f5830 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:13.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.890+0000 7fdb232c4700 1 -- 172.21.15.152:0/3662711916 >> 172.21.15.152:0/3662711916 conn(0x7fdb1c0f0b70 msgr2=0x7fdb1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:13.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.890+0000 7fdb232c4700 1 -- 172.21.15.152:0/3662711916 shutdown_connections 2023-12-06T03:57:13.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.890+0000 7fdb232c4700 1 -- 172.21.15.152:0/3662711916 wait complete. 2023-12-06T03:57:13.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.891+0000 7fdb232c4700 1 Processor -- start 2023-12-06T03:57:13.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.891+0000 7fdb232c4700 1 -- start start 2023-12-06T03:57:13.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.891+0000 7fdb232c4700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb1c0f5410 0x7fdb1c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:13.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.891+0000 7fdb232c4700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1c106c10 con 0x7fdb1c0f5410 2023-12-06T03:57:13.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.891+0000 7fdb21060700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb1c0f5410 0x7fdb1c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:13.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.891+0000 7fdb21060700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb1c0f5410 0x7fdb1c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:47792/0 (socket says 172.21.15.152:47792) 2023-12-06T03:57:13.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.891+0000 7fdb21060700 1 -- 172.21.15.152:0/699331267 learned_addr learned my addr 172.21.15.152:0/699331267 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:13.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.891+0000 7fdb21060700 1 -- 172.21.15.152:0/699331267 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb0c005040 con 0x7fdb1c0f5410 2023-12-06T03:57:13.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.892+0000 7fdb21060700 1 --2- 172.21.15.152:0/699331267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb1c0f5410 0x7fdb1c106720 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fdb0c014000 tx=0x7fdb0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:13.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.892+0000 7fdb11ffb700 1 -- 172.21.15.152:0/699331267 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb0c0095f0 con 0x7fdb1c0f5410 2023-12-06T03:57:13.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.892+0000 7fdb232c4700 1 -- 172.21.15.152:0/699331267 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1c106e10 con 0x7fdb1c0f5410 2023-12-06T03:57:13.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.892+0000 7fdb11ffb700 1 -- 172.21.15.152:0/699331267 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb0c007e70 con 0x7fdb1c0f5410 2023-12-06T03:57:13.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.892+0000 7fdb11ffb700 1 -- 172.21.15.152:0/699331267 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb0c014ea0 con 0x7fdb1c0f5410 2023-12-06T03:57:13.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.892+0000 7fdb232c4700 1 -- 172.21.15.152:0/699331267 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb1c1072a0 con 0x7fdb1c0f5410 2023-12-06T03:57:13.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.893+0000 7fdb11ffb700 1 -- 172.21.15.152:0/699331267 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb0c025070 con 0x7fdb1c0f5410 2023-12-06T03:57:13.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.893+0000 7fdb11ffb700 1 --2- 172.21.15.152:0/699331267 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdb0802ffb0 0x7fdb08032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:13.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.893+0000 7fdb232c4700 1 -- 172.21.15.152:0/699331267 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb1c045ad0 con 0x7fdb1c0f5410 2023-12-06T03:57:13.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.893+0000 7fdb11ffb700 1 -- 172.21.15.152:0/699331267 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb0c040a60 con 0x7fdb1c0f5410 2023-12-06T03:57:13.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.893+0000 7fdb2085f700 1 --2- 172.21.15.152:0/699331267 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdb0802ffb0 0x7fdb08032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:13.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.894+0000 7fdb2085f700 1 --2- 172.21.15.152:0/699331267 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdb0802ffb0 0x7fdb08032470 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fdb180096a0 tx=0x7fdb18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:13.899 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:13.898+0000 7fdb11ffb700 1 -- 172.21.15.152:0/699331267 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb0c00f710 con 0x7fdb1c0f5410 2023-12-06T03:57:14.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:14.102+0000 7fdb232c4700 1 -- 172.21.15.152:0/699331267 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdb1c108150 con 0x7fdb1c0f5410 2023-12-06T03:57:14.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:14.103+0000 7fdb11ffb700 1 -- 172.21.15.152:0/699331267 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdb0c012330 con 0x7fdb1c0f5410 2023-12-06T03:57:14.105 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:14.105 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:14.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:13 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:12.423293+0000 mgr.smithi132.gokxja (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:14.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:14.105+0000 7fdb232c4700 1 -- 172.21.15.152:0/699331267 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdb0802ffb0 msgr2=0x7fdb08032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:14.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:14.105+0000 7fdb232c4700 1 --2- 172.21.15.152:0/699331267 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdb0802ffb0 0x7fdb08032470 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7fdb180096a0 tx=0x7fdb18006b40).stop 2023-12-06T03:57:14.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:14.105+0000 7fdb232c4700 1 -- 172.21.15.152:0/699331267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb1c0f5410 msgr2=0x7fdb1c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:14.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:14.105+0000 7fdb232c4700 1 --2- 172.21.15.152:0/699331267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb1c0f5410 0x7fdb1c106720 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7fdb0c014000 tx=0x7fdb0c00b040).stop 2023-12-06T03:57:14.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:14.105+0000 7fdb232c4700 1 -- 172.21.15.152:0/699331267 shutdown_connections 2023-12-06T03:57:14.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:14.105+0000 7fdb232c4700 1 --2- 172.21.15.152:0/699331267 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdb0802ffb0 0x7fdb08032470 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:14.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:14.105+0000 7fdb232c4700 1 --2- 172.21.15.152:0/699331267 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdb1c0f5410 0x7fdb1c106720 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:14.108 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:14.105+0000 7fdb232c4700 1 -- 172.21.15.152:0/699331267 >> 172.21.15.152:0/699331267 conn(0x7fdb1c0f0b70 msgr2=0x7fdb1c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:14.108 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:14.105+0000 7fdb232c4700 1 -- 172.21.15.152:0/699331267 shutdown_connections 2023-12-06T03:57:14.108 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:14.105+0000 7fdb232c4700 1 -- 172.21.15.152:0/699331267 wait complete. 2023-12-06T03:57:14.108 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:15.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:14 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:14.104455+0000 mon.smithi132 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.152:0/699331267' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:15.466 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:15.467 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:16.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:15 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:14.423738+0000 mgr.smithi132.gokxja (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:16.637 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.635+0000 7fc1bffff700 1 -- 172.21.15.152:0/2767580277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1c00f51a0 msgr2=0x7fc1c00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:16.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.635+0000 7fc1bffff700 1 --2- 172.21.15.152:0/2767580277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1c00f51a0 0x7fc1c00f55c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7fc1b0009fb0 tx=0x7fc1b0009b30).stop 2023-12-06T03:57:16.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.636+0000 7fc1bffff700 1 -- 172.21.15.152:0/2767580277 shutdown_connections 2023-12-06T03:57:16.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.636+0000 7fc1bffff700 1 --2- 172.21.15.152:0/2767580277 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1c00f51a0 0x7fc1c00f55c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:16.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.636+0000 7fc1bffff700 1 -- 172.21.15.152:0/2767580277 >> 172.21.15.152:0/2767580277 conn(0x7fc1c00f09a0 msgr2=0x7fc1c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:16.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.636+0000 7fc1bffff700 1 -- 172.21.15.152:0/2767580277 shutdown_connections 2023-12-06T03:57:16.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.636+0000 7fc1bffff700 1 -- 172.21.15.152:0/2767580277 wait complete. 2023-12-06T03:57:16.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.636+0000 7fc1bffff700 1 Processor -- start 2023-12-06T03:57:16.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.636+0000 7fc1bffff700 1 -- start start 2023-12-06T03:57:16.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.636+0000 7fc1bffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1c00f51a0 0x7fc1c01091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:16.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.636+0000 7fc1bffff700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1c0060240 con 0x7fc1c00f51a0 2023-12-06T03:57:16.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.637+0000 7fc1beffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1c00f51a0 0x7fc1c01091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:16.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.637+0000 7fc1beffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1c00f51a0 0x7fc1c01091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:47812/0 (socket says 172.21.15.152:47812) 2023-12-06T03:57:16.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.637+0000 7fc1beffd700 1 -- 172.21.15.152:0/1891467139 learned_addr learned my addr 172.21.15.152:0/1891467139 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:16.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.637+0000 7fc1beffd700 1 -- 172.21.15.152:0/1891467139 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1b0005040 con 0x7fc1c00f51a0 2023-12-06T03:57:16.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.637+0000 7fc1beffd700 1 --2- 172.21.15.152:0/1891467139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1c00f51a0 0x7fc1c01091b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fc1b0013510 tx=0x7fc1b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:16.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.637+0000 7fc1a77fe700 1 -- 172.21.15.152:0/1891467139 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1b0009320 con 0x7fc1c00f51a0 2023-12-06T03:57:16.641 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.637+0000 7fc1bffff700 1 -- 172.21.15.152:0/1891467139 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1c0060440 con 0x7fc1c00f51a0 2023-12-06T03:57:16.641 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.637+0000 7fc1a77fe700 1 -- 172.21.15.152:0/1891467139 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc1b0013f00 con 0x7fc1c00f51a0 2023-12-06T03:57:16.641 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.637+0000 7fc1a77fe700 1 -- 172.21.15.152:0/1891467139 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1b0014dc0 con 0x7fc1c00f51a0 2023-12-06T03:57:16.641 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.637+0000 7fc1bffff700 1 -- 172.21.15.152:0/1891467139 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1c0109860 con 0x7fc1c00f51a0 2023-12-06T03:57:16.641 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.638+0000 7fc1a77fe700 1 -- 172.21.15.152:0/1891467139 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc1b00095f0 con 0x7fc1c00f51a0 2023-12-06T03:57:16.641 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.638+0000 7fc1bffff700 1 -- 172.21.15.152:0/1891467139 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1c0045ad0 con 0x7fc1c00f51a0 2023-12-06T03:57:16.641 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.638+0000 7fc1a77fe700 1 --2- 172.21.15.152:0/1891467139 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc1a802ff60 0x7fc1a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:16.641 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.638+0000 7fc1a77fe700 1 -- 172.21.15.152:0/1891467139 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc1b0041040 con 0x7fc1c00f51a0 2023-12-06T03:57:16.641 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.638+0000 7fc1be7fc700 1 --2- 172.21.15.152:0/1891467139 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc1a802ff60 0x7fc1a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:16.641 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.639+0000 7fc1be7fc700 1 --2- 172.21.15.152:0/1891467139 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc1a802ff60 0x7fc1a8032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fc1b40096a0 tx=0x7fc1b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:16.645 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.643+0000 7fc1a77fe700 1 -- 172.21.15.152:0/1891467139 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc1b0014420 con 0x7fc1c00f51a0 2023-12-06T03:57:16.850 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.848+0000 7fc1bffff700 1 -- 172.21.15.152:0/1891467139 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1c010a080 con 0x7fc1c00f51a0 2023-12-06T03:57:16.850 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.849+0000 7fc1a77fe700 1 -- 172.21.15.152:0/1891467139 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc1b0010230 con 0x7fc1c00f51a0 2023-12-06T03:57:16.851 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:16.851 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:16.852 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.850+0000 7fc1bffff700 1 -- 172.21.15.152:0/1891467139 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc1a802ff60 msgr2=0x7fc1a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:16.852 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.850+0000 7fc1bffff700 1 --2- 172.21.15.152:0/1891467139 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc1a802ff60 0x7fc1a8032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fc1b40096a0 tx=0x7fc1b4006b40).stop 2023-12-06T03:57:16.852 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.850+0000 7fc1bffff700 1 -- 172.21.15.152:0/1891467139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1c00f51a0 msgr2=0x7fc1c01091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:16.852 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.850+0000 7fc1bffff700 1 --2- 172.21.15.152:0/1891467139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1c00f51a0 0x7fc1c01091b0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fc1b0013510 tx=0x7fc1b000b040).stop 2023-12-06T03:57:16.852 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.850+0000 7fc1bffff700 1 -- 172.21.15.152:0/1891467139 shutdown_connections 2023-12-06T03:57:16.852 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.850+0000 7fc1bffff700 1 --2- 172.21.15.152:0/1891467139 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc1a802ff60 0x7fc1a8032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:16.853 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.850+0000 7fc1bffff700 1 --2- 172.21.15.152:0/1891467139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc1c00f51a0 0x7fc1c01091b0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:16.853 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.850+0000 7fc1bffff700 1 -- 172.21.15.152:0/1891467139 >> 172.21.15.152:0/1891467139 conn(0x7fc1c00f09a0 msgr2=0x7fc1c00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:16.853 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.850+0000 7fc1bffff700 1 -- 172.21.15.152:0/1891467139 shutdown_connections 2023-12-06T03:57:16.853 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:16.850+0000 7fc1bffff700 1 -- 172.21.15.152:0/1891467139 wait complete. 2023-12-06T03:57:16.853 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:18.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:17 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:16.424162+0000 mgr.smithi132.gokxja (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:18.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:17 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:16.850361+0000 mon.smithi132 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.152:0/1891467139' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:18.186 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:18.186 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:19.358 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.356+0000 7f24db87f700 1 -- 172.21.15.152:0/3196291647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24d40f6510 msgr2=0x7f24d40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:19.358 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.356+0000 7f24db87f700 1 --2- 172.21.15.152:0/3196291647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24d40f6510 0x7f24d40f6930 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f24c4009fb0 tx=0x7f24c4009b30).stop 2023-12-06T03:57:19.359 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.356+0000 7f24db87f700 1 -- 172.21.15.152:0/3196291647 shutdown_connections 2023-12-06T03:57:19.359 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.356+0000 7f24db87f700 1 --2- 172.21.15.152:0/3196291647 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24d40f6510 0x7f24d40f6930 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:19.359 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.356+0000 7f24db87f700 1 -- 172.21.15.152:0/3196291647 >> 172.21.15.152:0/3196291647 conn(0x7f24d40f1c70 msgr2=0x7f24d40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:19.359 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.356+0000 7f24db87f700 1 -- 172.21.15.152:0/3196291647 shutdown_connections 2023-12-06T03:57:19.359 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.356+0000 7f24db87f700 1 -- 172.21.15.152:0/3196291647 wait complete. 2023-12-06T03:57:19.359 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.357+0000 7f24db87f700 1 Processor -- start 2023-12-06T03:57:19.359 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.357+0000 7f24db87f700 1 -- start start 2023-12-06T03:57:19.359 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.357+0000 7f24db87f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24d40f6510 0x7f24d4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:19.359 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.357+0000 7f24db87f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24d4106c20 con 0x7f24d40f6510 2023-12-06T03:57:19.362 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.357+0000 7f24d961b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24d40f6510 0x7f24d4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:19.362 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.357+0000 7f24d961b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24d40f6510 0x7f24d4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:52386/0 (socket says 172.21.15.152:52386) 2023-12-06T03:57:19.362 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.357+0000 7f24d961b700 1 -- 172.21.15.152:0/1489203766 learned_addr learned my addr 172.21.15.152:0/1489203766 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:19.362 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.357+0000 7f24d961b700 1 -- 172.21.15.152:0/1489203766 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c4005040 con 0x7f24d40f6510 2023-12-06T03:57:19.362 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.357+0000 7f24d961b700 1 --2- 172.21.15.152:0/1489203766 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24d40f6510 0x7f24d4106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f24c4013510 tx=0x7f24c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:19.362 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.358+0000 7f24c9ffb700 1 -- 172.21.15.152:0/1489203766 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24c4009320 con 0x7f24d40f6510 2023-12-06T03:57:19.362 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.358+0000 7f24db87f700 1 -- 172.21.15.152:0/1489203766 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24d4106e20 con 0x7f24d40f6510 2023-12-06T03:57:19.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.358+0000 7f24c9ffb700 1 -- 172.21.15.152:0/1489203766 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f24c4013f00 con 0x7f24d40f6510 2023-12-06T03:57:19.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.358+0000 7f24c9ffb700 1 -- 172.21.15.152:0/1489203766 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f24c4014d90 con 0x7f24d40f6510 2023-12-06T03:57:19.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.358+0000 7f24db87f700 1 -- 172.21.15.152:0/1489203766 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24d41072b0 con 0x7f24d40f6510 2023-12-06T03:57:19.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.359+0000 7f24db87f700 1 -- 172.21.15.152:0/1489203766 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24d4045ad0 con 0x7f24d40f6510 2023-12-06T03:57:19.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.359+0000 7f24c9ffb700 1 -- 172.21.15.152:0/1489203766 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f24c40095f0 con 0x7f24d40f6510 2023-12-06T03:57:19.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.359+0000 7f24c9ffb700 1 --2- 172.21.15.152:0/1489203766 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f24c002ffb0 0x7f24c0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:19.363 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.359+0000 7f24c9ffb700 1 -- 172.21.15.152:0/1489203766 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f24c4040fe0 con 0x7f24d40f6510 2023-12-06T03:57:19.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.359+0000 7f24d8e1a700 1 --2- 172.21.15.152:0/1489203766 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f24c002ffb0 0x7f24c0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:19.364 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.359+0000 7f24d8e1a700 1 --2- 172.21.15.152:0/1489203766 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f24c002ffb0 0x7f24c0032470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f24d00096a0 tx=0x7f24d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:19.365 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.363+0000 7f24c9ffb700 1 -- 172.21.15.152:0/1489203766 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f24c406d2b0 con 0x7f24d40f6510 2023-12-06T03:57:19.571 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.569+0000 7f24db87f700 1 -- 172.21.15.152:0/1489203766 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24d40fbf60 con 0x7f24d40f6510 2023-12-06T03:57:19.571 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.569+0000 7f24c9ffb700 1 -- 172.21.15.152:0/1489203766 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f24c4014420 con 0x7f24d40f6510 2023-12-06T03:57:19.571 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:19.571 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:19.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.571+0000 7f24db87f700 1 -- 172.21.15.152:0/1489203766 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f24c002ffb0 msgr2=0x7f24c0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:19.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.571+0000 7f24db87f700 1 --2- 172.21.15.152:0/1489203766 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f24c002ffb0 0x7f24c0032470 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f24d00096a0 tx=0x7f24d0006b40).stop 2023-12-06T03:57:19.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.571+0000 7f24db87f700 1 -- 172.21.15.152:0/1489203766 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24d40f6510 msgr2=0x7f24d4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:19.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.571+0000 7f24db87f700 1 --2- 172.21.15.152:0/1489203766 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24d40f6510 0x7f24d4106730 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f24c4013510 tx=0x7f24c400b040).stop 2023-12-06T03:57:19.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.571+0000 7f24db87f700 1 -- 172.21.15.152:0/1489203766 shutdown_connections 2023-12-06T03:57:19.573 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.571+0000 7f24db87f700 1 --2- 172.21.15.152:0/1489203766 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f24c002ffb0 0x7f24c0032470 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:19.574 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.571+0000 7f24db87f700 1 --2- 172.21.15.152:0/1489203766 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f24d40f6510 0x7f24d4106730 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:19.574 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.571+0000 7f24db87f700 1 -- 172.21.15.152:0/1489203766 >> 172.21.15.152:0/1489203766 conn(0x7f24d40f1c70 msgr2=0x7f24d40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:19.574 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.571+0000 7f24db87f700 1 -- 172.21.15.152:0/1489203766 shutdown_connections 2023-12-06T03:57:19.574 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:19.571+0000 7f24db87f700 1 -- 172.21.15.152:0/1489203766 wait complete. 2023-12-06T03:57:19.574 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:20.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:19 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:18.424632+0000 mgr.smithi132.gokxja (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:20.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:19 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:19.570926+0000 mon.smithi132 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.152:0/1489203766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:20.961 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:20.961 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:22.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.095+0000 7f26a5a6e700 1 -- 172.21.15.152:0/3207161777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26a00f62a0 msgr2=0x7f26a00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:22.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.095+0000 7f26a5a6e700 1 --2- 172.21.15.152:0/3207161777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26a00f62a0 0x7f26a00f66c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f2694009fb0 tx=0x7f2694009b30).stop 2023-12-06T03:57:22.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.095+0000 7f26a5a6e700 1 -- 172.21.15.152:0/3207161777 shutdown_connections 2023-12-06T03:57:22.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.095+0000 7f26a5a6e700 1 --2- 172.21.15.152:0/3207161777 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26a00f62a0 0x7f26a00f66c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:22.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.095+0000 7f26a5a6e700 1 -- 172.21.15.152:0/3207161777 >> 172.21.15.152:0/3207161777 conn(0x7f26a00f1aa0 msgr2=0x7f26a00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:22.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.096+0000 7f26a5a6e700 1 -- 172.21.15.152:0/3207161777 shutdown_connections 2023-12-06T03:57:22.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.096+0000 7f26a5a6e700 1 -- 172.21.15.152:0/3207161777 wait complete. 2023-12-06T03:57:22.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.096+0000 7f26a5a6e700 1 Processor -- start 2023-12-06T03:57:22.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.096+0000 7f26a5a6e700 1 -- start start 2023-12-06T03:57:22.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.096+0000 7f26a5a6e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26a00f62a0 0x7f26a01086b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:22.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.096+0000 7f26a5a6e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a0108ba0 con 0x7f26a00f62a0 2023-12-06T03:57:22.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.097+0000 7f26a4a6c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26a00f62a0 0x7f26a01086b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:22.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.097+0000 7f26a4a6c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26a00f62a0 0x7f26a01086b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:52402/0 (socket says 172.21.15.152:52402) 2023-12-06T03:57:22.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.097+0000 7f26a4a6c700 1 -- 172.21.15.152:0/1719402211 learned_addr learned my addr 172.21.15.152:0/1719402211 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:22.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.097+0000 7f26a4a6c700 1 -- 172.21.15.152:0/1719402211 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2694005040 con 0x7f26a00f62a0 2023-12-06T03:57:22.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.097+0000 7f26a4a6c700 1 --2- 172.21.15.152:0/1719402211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26a00f62a0 0x7f26a01086b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f2694013510 tx=0x7f269400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:22.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.097+0000 7f269d7fa700 1 -- 172.21.15.152:0/1719402211 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2694009320 con 0x7f26a00f62a0 2023-12-06T03:57:22.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.097+0000 7f26a5a6e700 1 -- 172.21.15.152:0/1719402211 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a0108da0 con 0x7f26a00f62a0 2023-12-06T03:57:22.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.097+0000 7f269d7fa700 1 -- 172.21.15.152:0/1719402211 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2694013f00 con 0x7f26a00f62a0 2023-12-06T03:57:22.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.097+0000 7f269d7fa700 1 -- 172.21.15.152:0/1719402211 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2694014dc0 con 0x7f26a00f62a0 2023-12-06T03:57:22.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.097+0000 7f26a5a6e700 1 -- 172.21.15.152:0/1719402211 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a0109230 con 0x7f26a00f62a0 2023-12-06T03:57:22.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.098+0000 7f269d7fa700 1 -- 172.21.15.152:0/1719402211 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f269401d3d0 con 0x7f26a00f62a0 2023-12-06T03:57:22.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.098+0000 7f269d7fa700 1 --2- 172.21.15.152:0/1719402211 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f269002ff60 0x7f2690032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:22.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.098+0000 7f269d7fa700 1 -- 172.21.15.152:0/1719402211 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2694041120 con 0x7f26a00f62a0 2023-12-06T03:57:22.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.098+0000 7f26a5a6e700 1 -- 172.21.15.152:0/1719402211 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a0045ad0 con 0x7f26a00f62a0 2023-12-06T03:57:22.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.098+0000 7f269ffff700 1 --2- 172.21.15.152:0/1719402211 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f269002ff60 0x7f2690032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:22.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.099+0000 7f269ffff700 1 --2- 172.21.15.152:0/1719402211 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f269002ff60 0x7f2690032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f268c0096a0 tx=0x7f268c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:22.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.103+0000 7f269d7fa700 1 -- 172.21.15.152:0/1719402211 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2694012920 con 0x7f26a00f62a0 2023-12-06T03:57:22.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:20.425019+0000 mgr.smithi132.gokxja (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:22.312 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.310+0000 7f26a5a6e700 1 -- 172.21.15.152:0/1719402211 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26a0109f50 con 0x7f26a00f62a0 2023-12-06T03:57:22.313 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.311+0000 7f269d7fa700 1 -- 172.21.15.152:0/1719402211 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f269402b020 con 0x7f26a00f62a0 2023-12-06T03:57:22.313 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:22.313 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:22.314 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.313+0000 7f26a5a6e700 1 -- 172.21.15.152:0/1719402211 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f269002ff60 msgr2=0x7f2690032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:22.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.313+0000 7f26a5a6e700 1 --2- 172.21.15.152:0/1719402211 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f269002ff60 0x7f2690032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f268c0096a0 tx=0x7f268c006b40).stop 2023-12-06T03:57:22.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.313+0000 7f26a5a6e700 1 -- 172.21.15.152:0/1719402211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26a00f62a0 msgr2=0x7f26a01086b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:22.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.313+0000 7f26a5a6e700 1 --2- 172.21.15.152:0/1719402211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26a00f62a0 0x7f26a01086b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f2694013510 tx=0x7f269400b040).stop 2023-12-06T03:57:22.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.313+0000 7f26a5a6e700 1 -- 172.21.15.152:0/1719402211 shutdown_connections 2023-12-06T03:57:22.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.313+0000 7f26a5a6e700 1 --2- 172.21.15.152:0/1719402211 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f269002ff60 0x7f2690032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:22.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.313+0000 7f26a5a6e700 1 --2- 172.21.15.152:0/1719402211 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f26a00f62a0 0x7f26a01086b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:22.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.313+0000 7f26a5a6e700 1 -- 172.21.15.152:0/1719402211 >> 172.21.15.152:0/1719402211 conn(0x7f26a00f1aa0 msgr2=0x7f26a00f24d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:22.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.313+0000 7f26a5a6e700 1 -- 172.21.15.152:0/1719402211 shutdown_connections 2023-12-06T03:57:22.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:22.313+0000 7f26a5a6e700 1 -- 172.21.15.152:0/1719402211 wait complete. 2023-12-06T03:57:22.316 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:23.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:22 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:22.312448+0000 mon.smithi132 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.152:0/1719402211' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:23.703 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:23.703 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:24.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:22.425401+0000 mgr.smithi132.gokxja (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:24.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.859+0000 7fb80ab38700 1 -- 172.21.15.152:0/1171900702 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8040f7b60 msgr2=0x7fb8040f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:24.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.859+0000 7fb80ab38700 1 --2- 172.21.15.152:0/1171900702 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8040f7b60 0x7fb8040f7f80 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7fb7f8009fb0 tx=0x7fb7f8009b30).stop 2023-12-06T03:57:24.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.859+0000 7fb80ab38700 1 -- 172.21.15.152:0/1171900702 shutdown_connections 2023-12-06T03:57:24.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.859+0000 7fb80ab38700 1 --2- 172.21.15.152:0/1171900702 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8040f7b60 0x7fb8040f7f80 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:24.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.859+0000 7fb80ab38700 1 -- 172.21.15.152:0/1171900702 >> 172.21.15.152:0/1171900702 conn(0x7fb8040f02e0 msgr2=0x7fb8040f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:24.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.859+0000 7fb80ab38700 1 -- 172.21.15.152:0/1171900702 shutdown_connections 2023-12-06T03:57:24.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.859+0000 7fb80ab38700 1 -- 172.21.15.152:0/1171900702 wait complete. 2023-12-06T03:57:24.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.859+0000 7fb80ab38700 1 Processor -- start 2023-12-06T03:57:24.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.859+0000 7fb80ab38700 1 -- start start 2023-12-06T03:57:24.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.860+0000 7fb80ab38700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8040f7b60 0x7fb804106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:24.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.860+0000 7fb80ab38700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb804106c30 con 0x7fb8040f7b60 2023-12-06T03:57:24.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.860+0000 7fb8088d4700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8040f7b60 0x7fb804106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:24.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.860+0000 7fb8088d4700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8040f7b60 0x7fb804106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:52424/0 (socket says 172.21.15.152:52424) 2023-12-06T03:57:24.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.860+0000 7fb8088d4700 1 -- 172.21.15.152:0/2108106720 learned_addr learned my addr 172.21.15.152:0/2108106720 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:24.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.860+0000 7fb8088d4700 1 -- 172.21.15.152:0/2108106720 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f8005040 con 0x7fb8040f7b60 2023-12-06T03:57:24.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.860+0000 7fb8088d4700 1 --2- 172.21.15.152:0/2108106720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8040f7b60 0x7fb804106740 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fb7f8013510 tx=0x7fb7f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:24.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.860+0000 7fb8017fa700 1 -- 172.21.15.152:0/2108106720 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f8009320 con 0x7fb8040f7b60 2023-12-06T03:57:24.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.860+0000 7fb80ab38700 1 -- 172.21.15.152:0/2108106720 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb804106e30 con 0x7fb8040f7b60 2023-12-06T03:57:24.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.860+0000 7fb8017fa700 1 -- 172.21.15.152:0/2108106720 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7f8013f00 con 0x7fb8040f7b60 2023-12-06T03:57:24.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.860+0000 7fb8017fa700 1 -- 172.21.15.152:0/2108106720 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f8014d90 con 0x7fb8040f7b60 2023-12-06T03:57:24.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.860+0000 7fb80ab38700 1 -- 172.21.15.152:0/2108106720 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8041072c0 con 0x7fb8040f7b60 2023-12-06T03:57:24.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.861+0000 7fb8017fa700 1 -- 172.21.15.152:0/2108106720 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7f8014420 con 0x7fb8040f7b60 2023-12-06T03:57:24.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.861+0000 7fb80ab38700 1 -- 172.21.15.152:0/2108106720 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb804045ad0 con 0x7fb8040f7b60 2023-12-06T03:57:24.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.861+0000 7fb8017fa700 1 --2- 172.21.15.152:0/2108106720 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb7f402ff60 0x7fb7f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:24.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.861+0000 7fb8017fa700 1 -- 172.21.15.152:0/2108106720 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb7f8041110 con 0x7fb8040f7b60 2023-12-06T03:57:24.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.862+0000 7fb803fff700 1 --2- 172.21.15.152:0/2108106720 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb7f402ff60 0x7fb7f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:24.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.862+0000 7fb803fff700 1 --2- 172.21.15.152:0/2108106720 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb7f402ff60 0x7fb7f4032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb7f00096a0 tx=0x7fb7f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:24.867 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:24.866+0000 7fb8017fa700 1 -- 172.21.15.152:0/2108106720 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7f80095f0 con 0x7fb8040f7b60 2023-12-06T03:57:25.074 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:25.072+0000 7fb80ab38700 1 -- 172.21.15.152:0/2108106720 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb804108110 con 0x7fb8040f7b60 2023-12-06T03:57:25.074 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:25.073+0000 7fb8017fa700 1 -- 172.21.15.152:0/2108106720 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb7f801db70 con 0x7fb8040f7b60 2023-12-06T03:57:25.075 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:25.075 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:25.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:25.074+0000 7fb80ab38700 1 -- 172.21.15.152:0/2108106720 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb7f402ff60 msgr2=0x7fb7f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:25.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:25.074+0000 7fb80ab38700 1 --2- 172.21.15.152:0/2108106720 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb7f402ff60 0x7fb7f4032420 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fb7f00096a0 tx=0x7fb7f0006b40).stop 2023-12-06T03:57:25.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:25.074+0000 7fb80ab38700 1 -- 172.21.15.152:0/2108106720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8040f7b60 msgr2=0x7fb804106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:25.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:25.074+0000 7fb80ab38700 1 --2- 172.21.15.152:0/2108106720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8040f7b60 0x7fb804106740 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fb7f8013510 tx=0x7fb7f800b040).stop 2023-12-06T03:57:25.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:25.074+0000 7fb80ab38700 1 -- 172.21.15.152:0/2108106720 shutdown_connections 2023-12-06T03:57:25.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:25.074+0000 7fb80ab38700 1 --2- 172.21.15.152:0/2108106720 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb7f402ff60 0x7fb7f4032420 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:25.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:25.074+0000 7fb80ab38700 1 --2- 172.21.15.152:0/2108106720 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb8040f7b60 0x7fb804106740 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:25.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:25.074+0000 7fb80ab38700 1 -- 172.21.15.152:0/2108106720 >> 172.21.15.152:0/2108106720 conn(0x7fb8040f02e0 msgr2=0x7fb8040f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:25.076 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:25.074+0000 7fb80ab38700 1 -- 172.21.15.152:0/2108106720 shutdown_connections 2023-12-06T03:57:25.077 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:25.074+0000 7fb80ab38700 1 -- 172.21.15.152:0/2108106720 wait complete. 2023-12-06T03:57:25.077 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:26.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:24.425741+0000 mgr.smithi132.gokxja (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:26.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:25.074076+0000 mon.smithi132 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.152:0/2108106720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:26.440 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:26.440 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:27.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.484+0000 7f08f7fff700 1 -- 172.21.15.152:0/3328438316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f80f62a0 msgr2=0x7f08f80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:27.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.484+0000 7f08f7fff700 1 --2- 172.21.15.152:0/3328438316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f80f62a0 0x7f08f80f66c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f08e8009fb0 tx=0x7f08e8009b30).stop 2023-12-06T03:57:27.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.484+0000 7f08f7fff700 1 -- 172.21.15.152:0/3328438316 shutdown_connections 2023-12-06T03:57:27.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.484+0000 7f08f7fff700 1 --2- 172.21.15.152:0/3328438316 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f80f62a0 0x7f08f80f66c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:27.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.484+0000 7f08f7fff700 1 -- 172.21.15.152:0/3328438316 >> 172.21.15.152:0/3328438316 conn(0x7f08f80f1ac0 msgr2=0x7f08f80f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:27.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.484+0000 7f08f7fff700 1 -- 172.21.15.152:0/3328438316 shutdown_connections 2023-12-06T03:57:27.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.484+0000 7f08f7fff700 1 -- 172.21.15.152:0/3328438316 wait complete. 2023-12-06T03:57:27.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.484+0000 7f08f7fff700 1 Processor -- start 2023-12-06T03:57:27.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.484+0000 7f08f7fff700 1 -- start start 2023-12-06T03:57:27.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.485+0000 7f08f7fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f80f62a0 0x7f08f81064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:27.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.485+0000 7f08f7fff700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08f8106990 con 0x7f08f80f62a0 2023-12-06T03:57:27.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.485+0000 7f08f6ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f80f62a0 0x7f08f81064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:27.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.485+0000 7f08f6ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f80f62a0 0x7f08f81064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:52436/0 (socket says 172.21.15.152:52436) 2023-12-06T03:57:27.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.485+0000 7f08f6ffd700 1 -- 172.21.15.152:0/1837849309 learned_addr learned my addr 172.21.15.152:0/1837849309 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:27.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.485+0000 7f08f6ffd700 1 -- 172.21.15.152:0/1837849309 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08e8005040 con 0x7f08f80f62a0 2023-12-06T03:57:27.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.485+0000 7f08f6ffd700 1 --2- 172.21.15.152:0/1837849309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f80f62a0 0x7f08f81064a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f08e8009830 tx=0x7f08e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:27.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.485+0000 7f08df7fe700 1 -- 172.21.15.152:0/1837849309 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08e8009320 con 0x7f08f80f62a0 2023-12-06T03:57:27.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.485+0000 7f08f7fff700 1 -- 172.21.15.152:0/1837849309 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08f8106b90 con 0x7f08f80f62a0 2023-12-06T03:57:27.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.485+0000 7f08df7fe700 1 -- 172.21.15.152:0/1837849309 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08e80136c0 con 0x7f08f80f62a0 2023-12-06T03:57:27.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.485+0000 7f08df7fe700 1 -- 172.21.15.152:0/1837849309 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08e801ee90 con 0x7f08f80f62a0 2023-12-06T03:57:27.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.485+0000 7f08f7fff700 1 -- 172.21.15.152:0/1837849309 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08f8107020 con 0x7f08f80f62a0 2023-12-06T03:57:27.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.486+0000 7f08f7fff700 1 -- 172.21.15.152:0/1837849309 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08f8045ad0 con 0x7f08f80f62a0 2023-12-06T03:57:27.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.486+0000 7f08df7fe700 1 -- 172.21.15.152:0/1837849309 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f08e801e420 con 0x7f08f80f62a0 2023-12-06T03:57:27.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.486+0000 7f08df7fe700 1 --2- 172.21.15.152:0/1837849309 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f08e002ff10 0x7f08e00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:27.490 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.487+0000 7f08df7fe700 1 -- 172.21.15.152:0/1837849309 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f08e8040f40 con 0x7f08f80f62a0 2023-12-06T03:57:27.490 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.487+0000 7f08f67fc700 1 --2- 172.21.15.152:0/1837849309 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f08e002ff10 0x7f08e00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:27.490 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.487+0000 7f08f67fc700 1 --2- 172.21.15.152:0/1837849309 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f08e002ff10 0x7f08e00323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f08ec0096a0 tx=0x7f08ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:27.493 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.492+0000 7f08df7fe700 1 -- 172.21.15.152:0/1837849309 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08e806d2b0 con 0x7f08f80f62a0 2023-12-06T03:57:27.701 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.699+0000 7f08f7fff700 1 -- 172.21.15.152:0/1837849309 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08f8107e70 con 0x7f08f80f62a0 2023-12-06T03:57:27.701 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.701+0000 7f08df7fe700 1 -- 172.21.15.152:0/1837849309 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f08e80095f0 con 0x7f08f80f62a0 2023-12-06T03:57:27.702 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:27.702 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:27.703 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.702+0000 7f08f7fff700 1 -- 172.21.15.152:0/1837849309 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f08e002ff10 msgr2=0x7f08e00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:27.703 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.702+0000 7f08f7fff700 1 --2- 172.21.15.152:0/1837849309 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f08e002ff10 0x7f08e00323d0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f08ec0096a0 tx=0x7f08ec006b40).stop 2023-12-06T03:57:27.704 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.702+0000 7f08f7fff700 1 -- 172.21.15.152:0/1837849309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f80f62a0 msgr2=0x7f08f81064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:27.704 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.702+0000 7f08f7fff700 1 --2- 172.21.15.152:0/1837849309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f80f62a0 0x7f08f81064a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f08e8009830 tx=0x7f08e800b040).stop 2023-12-06T03:57:27.704 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.702+0000 7f08f7fff700 1 -- 172.21.15.152:0/1837849309 shutdown_connections 2023-12-06T03:57:27.704 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.702+0000 7f08f7fff700 1 --2- 172.21.15.152:0/1837849309 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f08e002ff10 0x7f08e00323d0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:27.704 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.702+0000 7f08f7fff700 1 --2- 172.21.15.152:0/1837849309 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f08f80f62a0 0x7f08f81064a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:27.704 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.702+0000 7f08f7fff700 1 -- 172.21.15.152:0/1837849309 >> 172.21.15.152:0/1837849309 conn(0x7f08f80f1ac0 msgr2=0x7f08f80f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:27.704 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.702+0000 7f08f7fff700 1 -- 172.21.15.152:0/1837849309 shutdown_connections 2023-12-06T03:57:27.704 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:27.702+0000 7f08f7fff700 1 -- 172.21.15.152:0/1837849309 wait complete. 2023-12-06T03:57:27.704 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:28.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:26.425999+0000 mgr.smithi132.gokxja (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:29.069 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:29.070 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:29.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:28 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:27.701346+0000 mon.smithi132 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.152:0/1837849309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:30.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:28.426319+0000 mgr.smithi132.gokxja (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:30.142 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.139+0000 7eff8b59e700 1 -- 172.21.15.152:0/257049142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff8c0f62a0 msgr2=0x7eff8c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:30.142 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.139+0000 7eff8b59e700 1 --2- 172.21.15.152:0/257049142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff8c0f62a0 0x7eff8c0f66c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7eff7c009fb0 tx=0x7eff7c009b30).stop 2023-12-06T03:57:30.142 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.139+0000 7eff8b59e700 1 -- 172.21.15.152:0/257049142 shutdown_connections 2023-12-06T03:57:30.142 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.139+0000 7eff8b59e700 1 --2- 172.21.15.152:0/257049142 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff8c0f62a0 0x7eff8c0f66c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:30.142 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.139+0000 7eff8b59e700 1 -- 172.21.15.152:0/257049142 >> 172.21.15.152:0/257049142 conn(0x7eff8c0f1ac0 msgr2=0x7eff8c0f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:30.142 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.139+0000 7eff8b59e700 1 -- 172.21.15.152:0/257049142 shutdown_connections 2023-12-06T03:57:30.142 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.139+0000 7eff8b59e700 1 -- 172.21.15.152:0/257049142 wait complete. 2023-12-06T03:57:30.143 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.139+0000 7eff8b59e700 1 Processor -- start 2023-12-06T03:57:30.143 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.139+0000 7eff8b59e700 1 -- start start 2023-12-06T03:57:30.143 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.139+0000 7eff8b59e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff8c0f62a0 0x7eff8c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:30.143 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.139+0000 7eff8b59e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8c069e20 con 0x7eff8c0f62a0 2023-12-06T03:57:30.143 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.140+0000 7eff8a59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff8c0f62a0 0x7eff8c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:30.143 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.140+0000 7eff8a59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff8c0f62a0 0x7eff8c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:50650/0 (socket says 172.21.15.152:50650) 2023-12-06T03:57:30.143 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.140+0000 7eff8a59c700 1 -- 172.21.15.152:0/3873560307 learned_addr learned my addr 172.21.15.152:0/3873560307 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:30.143 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.140+0000 7eff8a59c700 1 -- 172.21.15.152:0/3873560307 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff7c005040 con 0x7eff8c0f62a0 2023-12-06T03:57:30.143 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.140+0000 7eff8a59c700 1 --2- 172.21.15.152:0/3873560307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff8c0f62a0 0x7eff8c06b9a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7eff7c013510 tx=0x7eff7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:30.143 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.140+0000 7eff7affd700 1 -- 172.21.15.152:0/3873560307 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff7c009320 con 0x7eff8c0f62a0 2023-12-06T03:57:30.143 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.140+0000 7eff8b59e700 1 -- 172.21.15.152:0/3873560307 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff8c069fc0 con 0x7eff8c0f62a0 2023-12-06T03:57:30.143 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.140+0000 7eff7affd700 1 -- 172.21.15.152:0/3873560307 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff7c013f00 con 0x7eff8c0f62a0 2023-12-06T03:57:30.143 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.140+0000 7eff7affd700 1 -- 172.21.15.152:0/3873560307 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff7c014dc0 con 0x7eff8c0f62a0 2023-12-06T03:57:30.143 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.140+0000 7eff8b59e700 1 -- 172.21.15.152:0/3873560307 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8c06a450 con 0x7eff8c0f62a0 2023-12-06T03:57:30.144 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.141+0000 7eff7affd700 1 -- 172.21.15.152:0/3873560307 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7eff7c01d3d0 con 0x7eff8c0f62a0 2023-12-06T03:57:30.144 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.141+0000 7eff8b59e700 1 -- 172.21.15.152:0/3873560307 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff8c059100 con 0x7eff8c0f62a0 2023-12-06T03:57:30.144 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.141+0000 7eff7affd700 1 --2- 172.21.15.152:0/3873560307 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7eff7402ff60 0x7eff74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:30.145 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.141+0000 7eff7affd700 1 -- 172.21.15.152:0/3873560307 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7eff7c041030 con 0x7eff8c0f62a0 2023-12-06T03:57:30.145 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.142+0000 7eff89d9b700 1 --2- 172.21.15.152:0/3873560307 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7eff7402ff60 0x7eff74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:30.145 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.142+0000 7eff89d9b700 1 --2- 172.21.15.152:0/3873560307 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7eff7402ff60 0x7eff74032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7eff800096a0 tx=0x7eff80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:30.148 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.146+0000 7eff7affd700 1 -- 172.21.15.152:0/3873560307 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff7c0095f0 con 0x7eff8c0f62a0 2023-12-06T03:57:30.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.353+0000 7eff8b59e700 1 -- 172.21.15.152:0/3873560307 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff8c0fbce0 con 0x7eff8c0f62a0 2023-12-06T03:57:30.355 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.355+0000 7eff7affd700 1 -- 172.21.15.152:0/3873560307 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7eff7c01c970 con 0x7eff8c0f62a0 2023-12-06T03:57:30.356 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:30.356 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:30.357 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.356+0000 7eff8b59e700 1 -- 172.21.15.152:0/3873560307 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7eff7402ff60 msgr2=0x7eff74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:30.358 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.356+0000 7eff8b59e700 1 --2- 172.21.15.152:0/3873560307 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7eff7402ff60 0x7eff74032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7eff800096a0 tx=0x7eff80006b40).stop 2023-12-06T03:57:30.358 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.356+0000 7eff8b59e700 1 -- 172.21.15.152:0/3873560307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff8c0f62a0 msgr2=0x7eff8c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:30.358 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.356+0000 7eff8b59e700 1 --2- 172.21.15.152:0/3873560307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff8c0f62a0 0x7eff8c06b9a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7eff7c013510 tx=0x7eff7c00b040).stop 2023-12-06T03:57:30.358 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.356+0000 7eff8b59e700 1 -- 172.21.15.152:0/3873560307 shutdown_connections 2023-12-06T03:57:30.358 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.356+0000 7eff8b59e700 1 --2- 172.21.15.152:0/3873560307 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7eff7402ff60 0x7eff74032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:30.358 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.356+0000 7eff8b59e700 1 --2- 172.21.15.152:0/3873560307 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7eff8c0f62a0 0x7eff8c06b9a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:30.358 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.356+0000 7eff8b59e700 1 -- 172.21.15.152:0/3873560307 >> 172.21.15.152:0/3873560307 conn(0x7eff8c0f1ac0 msgr2=0x7eff8c0f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:30.358 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.356+0000 7eff8b59e700 1 -- 172.21.15.152:0/3873560307 shutdown_connections 2023-12-06T03:57:30.358 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:30.356+0000 7eff8b59e700 1 -- 172.21.15.152:0/3873560307 wait complete. 2023-12-06T03:57:30.358 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:31.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:30 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:30.355253+0000 mon.smithi132 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.152:0/3873560307' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:31.713 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:31.713 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:32.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:31 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:30.426638+0000 mgr.smithi132.gokxja (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:32.797 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.796+0000 7ff1ec032700 1 -- 172.21.15.152:0/1745143963 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e40f6510 msgr2=0x7ff1e40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:32.798 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.796+0000 7ff1ec032700 1 --2- 172.21.15.152:0/1745143963 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e40f6510 0x7ff1e40f6930 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7ff1d4009fb0 tx=0x7ff1d4009b30).stop 2023-12-06T03:57:32.798 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.796+0000 7ff1ec032700 1 -- 172.21.15.152:0/1745143963 shutdown_connections 2023-12-06T03:57:32.798 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.796+0000 7ff1ec032700 1 --2- 172.21.15.152:0/1745143963 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e40f6510 0x7ff1e40f6930 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:32.798 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.796+0000 7ff1ec032700 1 -- 172.21.15.152:0/1745143963 >> 172.21.15.152:0/1745143963 conn(0x7ff1e40f1c70 msgr2=0x7ff1e40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:32.798 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.796+0000 7ff1ec032700 1 -- 172.21.15.152:0/1745143963 shutdown_connections 2023-12-06T03:57:32.798 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.796+0000 7ff1ec032700 1 -- 172.21.15.152:0/1745143963 wait complete. 2023-12-06T03:57:32.798 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.797+0000 7ff1ec032700 1 Processor -- start 2023-12-06T03:57:32.799 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.797+0000 7ff1ec032700 1 -- start start 2023-12-06T03:57:32.800 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.797+0000 7ff1ec032700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e40f6510 0x7ff1e4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:32.800 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.797+0000 7ff1ec032700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1e4106c20 con 0x7ff1e40f6510 2023-12-06T03:57:32.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.797+0000 7ff1e9dce700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e40f6510 0x7ff1e4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:32.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.797+0000 7ff1e9dce700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e40f6510 0x7ff1e4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:50676/0 (socket says 172.21.15.152:50676) 2023-12-06T03:57:32.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.797+0000 7ff1e9dce700 1 -- 172.21.15.152:0/4044310697 learned_addr learned my addr 172.21.15.152:0/4044310697 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:32.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.798+0000 7ff1e9dce700 1 -- 172.21.15.152:0/4044310697 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d4005040 con 0x7ff1e40f6510 2023-12-06T03:57:32.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.798+0000 7ff1e9dce700 1 --2- 172.21.15.152:0/4044310697 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e40f6510 0x7ff1e4106730 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ff1d4014000 tx=0x7ff1d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:32.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.798+0000 7ff1da7fc700 1 -- 172.21.15.152:0/4044310697 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1d40095f0 con 0x7ff1e40f6510 2023-12-06T03:57:32.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.798+0000 7ff1da7fc700 1 -- 172.21.15.152:0/4044310697 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff1d4007e70 con 0x7ff1e40f6510 2023-12-06T03:57:32.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.798+0000 7ff1ec032700 1 -- 172.21.15.152:0/4044310697 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1e4106e20 con 0x7ff1e40f6510 2023-12-06T03:57:32.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.798+0000 7ff1da7fc700 1 -- 172.21.15.152:0/4044310697 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1d4014ea0 con 0x7ff1e40f6510 2023-12-06T03:57:32.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.798+0000 7ff1ec032700 1 -- 172.21.15.152:0/4044310697 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1e41072b0 con 0x7ff1e40f6510 2023-12-06T03:57:32.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.799+0000 7ff1da7fc700 1 -- 172.21.15.152:0/4044310697 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff1d4025070 con 0x7ff1e40f6510 2023-12-06T03:57:32.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.799+0000 7ff1ec032700 1 -- 172.21.15.152:0/4044310697 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1e4045ad0 con 0x7ff1e40f6510 2023-12-06T03:57:32.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.799+0000 7ff1da7fc700 1 --2- 172.21.15.152:0/4044310697 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff1d002ff60 0x7ff1d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:32.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.799+0000 7ff1da7fc700 1 -- 172.21.15.152:0/4044310697 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff1d4040a30 con 0x7ff1e40f6510 2023-12-06T03:57:32.802 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.799+0000 7ff1e95cd700 1 --2- 172.21.15.152:0/4044310697 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff1d002ff60 0x7ff1d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:32.802 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.800+0000 7ff1e95cd700 1 --2- 172.21.15.152:0/4044310697 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff1d002ff60 0x7ff1d0032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7ff1e00096a0 tx=0x7ff1e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:32.804 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:32.804+0000 7ff1da7fc700 1 -- 172.21.15.152:0/4044310697 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff1d400f4e0 con 0x7ff1e40f6510 2023-12-06T03:57:33.011 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:33.010+0000 7ff1ec032700 1 -- 172.21.15.152:0/4044310697 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1e40fbf60 con 0x7ff1e40f6510 2023-12-06T03:57:33.012 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:33.011+0000 7ff1da7fc700 1 -- 172.21.15.152:0/4044310697 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff1d400f300 con 0x7ff1e40f6510 2023-12-06T03:57:33.012 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:33.013 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:33.013 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:33.013+0000 7ff1ec032700 1 -- 172.21.15.152:0/4044310697 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff1d002ff60 msgr2=0x7ff1d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:33.013 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:33.013+0000 7ff1ec032700 1 --2- 172.21.15.152:0/4044310697 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff1d002ff60 0x7ff1d0032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7ff1e00096a0 tx=0x7ff1e0006b40).stop 2023-12-06T03:57:33.013 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:33.013+0000 7ff1ec032700 1 -- 172.21.15.152:0/4044310697 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e40f6510 msgr2=0x7ff1e4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:33.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:33.013+0000 7ff1ec032700 1 --2- 172.21.15.152:0/4044310697 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e40f6510 0x7ff1e4106730 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ff1d4014000 tx=0x7ff1d400b040).stop 2023-12-06T03:57:33.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:33.013+0000 7ff1ec032700 1 -- 172.21.15.152:0/4044310697 shutdown_connections 2023-12-06T03:57:33.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:33.013+0000 7ff1ec032700 1 --2- 172.21.15.152:0/4044310697 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff1d002ff60 0x7ff1d0032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:33.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:33.013+0000 7ff1ec032700 1 --2- 172.21.15.152:0/4044310697 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1e40f6510 0x7ff1e4106730 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:33.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:33.013+0000 7ff1ec032700 1 -- 172.21.15.152:0/4044310697 >> 172.21.15.152:0/4044310697 conn(0x7ff1e40f1c70 msgr2=0x7ff1e40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:33.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:33.013+0000 7ff1ec032700 1 -- 172.21.15.152:0/4044310697 shutdown_connections 2023-12-06T03:57:33.014 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:33.013+0000 7ff1ec032700 1 -- 172.21.15.152:0/4044310697 wait complete. 2023-12-06T03:57:33.014 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:34.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:33 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:32.427017+0000 mgr.smithi132.gokxja (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:34.110 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:33 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:33.011406+0000 mon.smithi132 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.152:0/4044310697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:34.401 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:34.402 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:35.551 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.549+0000 7f68a5f87700 1 -- 172.21.15.152:0/2849861942 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68a0060280 msgr2=0x7f68a00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:35.551 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.549+0000 7f68a5f87700 1 --2- 172.21.15.152:0/2849861942 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68a0060280 0x7f68a00f90f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f6894009fb0 tx=0x7f6894009b30).stop 2023-12-06T03:57:35.551 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.549+0000 7f68a5f87700 1 -- 172.21.15.152:0/2849861942 shutdown_connections 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.549+0000 7f68a5f87700 1 --2- 172.21.15.152:0/2849861942 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68a0060280 0x7f68a00f90f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.549+0000 7f68a5f87700 1 -- 172.21.15.152:0/2849861942 >> 172.21.15.152:0/2849861942 conn(0x7f68a00f0b70 msgr2=0x7f68a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.549+0000 7f68a5f87700 1 -- 172.21.15.152:0/2849861942 shutdown_connections 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.549+0000 7f68a5f87700 1 -- 172.21.15.152:0/2849861942 wait complete. 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.549+0000 7f68a5f87700 1 Processor -- start 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.550+0000 7f68a5f87700 1 -- start start 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.550+0000 7f68a5f87700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68a0060280 0x7f68a00f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.550+0000 7f68a5f87700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68a00f4ff0 con 0x7f68a0060280 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.550+0000 7f689f7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68a0060280 0x7f68a00f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.550+0000 7f689f7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68a0060280 0x7f68a00f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:50702/0 (socket says 172.21.15.152:50702) 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.550+0000 7f689f7fe700 1 -- 172.21.15.152:0/3499232541 learned_addr learned my addr 172.21.15.152:0/3499232541 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.550+0000 7f689f7fe700 1 -- 172.21.15.152:0/3499232541 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6894005040 con 0x7f68a0060280 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.550+0000 7f689f7fe700 1 --2- 172.21.15.152:0/3499232541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68a0060280 0x7f68a00f6b70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f6894013510 tx=0x7f689400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.550+0000 7f6887fff700 1 -- 172.21.15.152:0/3499232541 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6894009320 con 0x7f68a0060280 2023-12-06T03:57:35.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.550+0000 7f68a5f87700 1 -- 172.21.15.152:0/3499232541 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68a00f5190 con 0x7f68a0060280 2023-12-06T03:57:35.553 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.550+0000 7f6887fff700 1 -- 172.21.15.152:0/3499232541 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6894013f00 con 0x7f68a0060280 2023-12-06T03:57:35.553 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.550+0000 7f6887fff700 1 -- 172.21.15.152:0/3499232541 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6894014dc0 con 0x7f68a0060280 2023-12-06T03:57:35.553 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.550+0000 7f68a5f87700 1 -- 172.21.15.152:0/3499232541 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68a00f5620 con 0x7f68a0060280 2023-12-06T03:57:35.553 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.551+0000 7f6887fff700 1 -- 172.21.15.152:0/3499232541 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6894014420 con 0x7f68a0060280 2023-12-06T03:57:35.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.551+0000 7f68a5f87700 1 -- 172.21.15.152:0/3499232541 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68a0059100 con 0x7f68a0060280 2023-12-06T03:57:35.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.551+0000 7f6887fff700 1 --2- 172.21.15.152:0/3499232541 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f688802ff60 0x7f6888032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:35.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.551+0000 7f6887fff700 1 -- 172.21.15.152:0/3499232541 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6894041050 con 0x7f68a0060280 2023-12-06T03:57:35.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.552+0000 7f689effd700 1 --2- 172.21.15.152:0/3499232541 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f688802ff60 0x7f6888032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:35.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.552+0000 7f689effd700 1 --2- 172.21.15.152:0/3499232541 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f688802ff60 0x7f6888032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f68900096a0 tx=0x7f6890006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:35.557 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.556+0000 7f6887fff700 1 -- 172.21.15.152:0/3499232541 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68940095f0 con 0x7f68a0060280 2023-12-06T03:57:35.765 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.764+0000 7f68a5f87700 1 -- 172.21.15.152:0/3499232541 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68a00fbf50 con 0x7f68a0060280 2023-12-06T03:57:35.765 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.764+0000 7f6887fff700 1 -- 172.21.15.152:0/3499232541 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f689401cb10 con 0x7f68a0060280 2023-12-06T03:57:35.765 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:35.766 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:35.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.766+0000 7f68a5f87700 1 -- 172.21.15.152:0/3499232541 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f688802ff60 msgr2=0x7f6888032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:35.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.766+0000 7f68a5f87700 1 --2- 172.21.15.152:0/3499232541 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f688802ff60 0x7f6888032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f68900096a0 tx=0x7f6890006b40).stop 2023-12-06T03:57:35.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.766+0000 7f68a5f87700 1 -- 172.21.15.152:0/3499232541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68a0060280 msgr2=0x7f68a00f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:35.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.766+0000 7f68a5f87700 1 --2- 172.21.15.152:0/3499232541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68a0060280 0x7f68a00f6b70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f6894013510 tx=0x7f689400b040).stop 2023-12-06T03:57:35.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.766+0000 7f68a5f87700 1 -- 172.21.15.152:0/3499232541 shutdown_connections 2023-12-06T03:57:35.768 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.766+0000 7f68a5f87700 1 --2- 172.21.15.152:0/3499232541 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f688802ff60 0x7f6888032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:35.768 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.766+0000 7f68a5f87700 1 --2- 172.21.15.152:0/3499232541 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f68a0060280 0x7f68a00f6b70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:35.768 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.766+0000 7f68a5f87700 1 -- 172.21.15.152:0/3499232541 >> 172.21.15.152:0/3499232541 conn(0x7f68a00f0b70 msgr2=0x7f68a00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:35.768 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.766+0000 7f68a5f87700 1 -- 172.21.15.152:0/3499232541 shutdown_connections 2023-12-06T03:57:35.768 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:35.766+0000 7f68a5f87700 1 -- 172.21.15.152:0/3499232541 wait complete. 2023-12-06T03:57:35.768 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:36.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:35 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:34.427401+0000 mgr.smithi132.gokxja (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:37.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:36 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:35.764895+0000 mon.smithi132 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.152:0/3499232541' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:37.155 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:37.155 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:38.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:36.427752+0000 mgr.smithi132.gokxja (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:38.286 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.282+0000 7fef2165a700 1 -- 172.21.15.152:0/4086361210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef1c0f5410 msgr2=0x7fef1c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:38.286 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.282+0000 7fef2165a700 1 --2- 172.21.15.152:0/4086361210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef1c0f5410 0x7fef1c0f5830 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fef10009fb0 tx=0x7fef10009b30).stop 2023-12-06T03:57:38.286 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.282+0000 7fef2165a700 1 -- 172.21.15.152:0/4086361210 shutdown_connections 2023-12-06T03:57:38.286 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.282+0000 7fef2165a700 1 --2- 172.21.15.152:0/4086361210 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef1c0f5410 0x7fef1c0f5830 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:38.286 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.282+0000 7fef2165a700 1 -- 172.21.15.152:0/4086361210 >> 172.21.15.152:0/4086361210 conn(0x7fef1c0f0b70 msgr2=0x7fef1c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:38.286 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.282+0000 7fef2165a700 1 -- 172.21.15.152:0/4086361210 shutdown_connections 2023-12-06T03:57:38.286 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.282+0000 7fef2165a700 1 -- 172.21.15.152:0/4086361210 wait complete. 2023-12-06T03:57:38.286 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.283+0000 7fef2165a700 1 Processor -- start 2023-12-06T03:57:38.286 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.283+0000 7fef2165a700 1 -- start start 2023-12-06T03:57:38.286 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.283+0000 7fef2165a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef1c0f5410 0x7fef1c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:38.286 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.283+0000 7fef2165a700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef1c105b20 con 0x7fef1c0f5410 2023-12-06T03:57:38.286 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.283+0000 7fef1affd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef1c0f5410 0x7fef1c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:38.287 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.283+0000 7fef1affd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef1c0f5410 0x7fef1c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:50722/0 (socket says 172.21.15.152:50722) 2023-12-06T03:57:38.287 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.283+0000 7fef1affd700 1 -- 172.21.15.152:0/3050543192 learned_addr learned my addr 172.21.15.152:0/3050543192 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:38.287 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.284+0000 7fef1affd700 1 -- 172.21.15.152:0/3050543192 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef10005040 con 0x7fef1c0f5410 2023-12-06T03:57:38.287 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.284+0000 7fef1affd700 1 --2- 172.21.15.152:0/3050543192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef1c0f5410 0x7fef1c105630 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fef10013510 tx=0x7fef1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:38.287 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.284+0000 7fef037fe700 1 -- 172.21.15.152:0/3050543192 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef10009320 con 0x7fef1c0f5410 2023-12-06T03:57:38.287 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.284+0000 7fef2165a700 1 -- 172.21.15.152:0/3050543192 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef1c105d20 con 0x7fef1c0f5410 2023-12-06T03:57:38.287 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.284+0000 7fef037fe700 1 -- 172.21.15.152:0/3050543192 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef10013f00 con 0x7fef1c0f5410 2023-12-06T03:57:38.287 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.284+0000 7fef037fe700 1 -- 172.21.15.152:0/3050543192 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef10014d90 con 0x7fef1c0f5410 2023-12-06T03:57:38.287 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.284+0000 7fef2165a700 1 -- 172.21.15.152:0/3050543192 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1c1061b0 con 0x7fef1c0f5410 2023-12-06T03:57:38.287 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.285+0000 7fef037fe700 1 -- 172.21.15.152:0/3050543192 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef100263a0 con 0x7fef1c0f5410 2023-12-06T03:57:38.287 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.285+0000 7fef037fe700 1 --2- 172.21.15.152:0/3050543192 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fef0402ff60 0x7fef04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:38.287 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.285+0000 7fef2165a700 1 -- 172.21.15.152:0/3050543192 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef1c045ad0 con 0x7fef1c0f5410 2023-12-06T03:57:38.287 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.285+0000 7fef037fe700 1 -- 172.21.15.152:0/3050543192 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef10041370 con 0x7fef1c0f5410 2023-12-06T03:57:38.287 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.285+0000 7fef1a7fc700 1 --2- 172.21.15.152:0/3050543192 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fef0402ff60 0x7fef04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:38.288 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.285+0000 7fef1a7fc700 1 --2- 172.21.15.152:0/3050543192 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fef0402ff60 0x7fef04032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fef0c0096a0 tx=0x7fef0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:38.290 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.290+0000 7fef037fe700 1 -- 172.21.15.152:0/3050543192 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef10012330 con 0x7fef1c0f5410 2023-12-06T03:57:38.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.495+0000 7fef2165a700 1 -- 172.21.15.152:0/3050543192 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef1c0faf50 con 0x7fef1c0f5410 2023-12-06T03:57:38.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.495+0000 7fef037fe700 1 -- 172.21.15.152:0/3050543192 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fef10012330 con 0x7fef1c0f5410 2023-12-06T03:57:38.497 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:38.497 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:38.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.497+0000 7fef2165a700 1 -- 172.21.15.152:0/3050543192 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fef0402ff60 msgr2=0x7fef04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:38.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.497+0000 7fef2165a700 1 --2- 172.21.15.152:0/3050543192 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fef0402ff60 0x7fef04032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fef0c0096a0 tx=0x7fef0c006b40).stop 2023-12-06T03:57:38.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.497+0000 7fef2165a700 1 -- 172.21.15.152:0/3050543192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef1c0f5410 msgr2=0x7fef1c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:38.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.497+0000 7fef2165a700 1 --2- 172.21.15.152:0/3050543192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef1c0f5410 0x7fef1c105630 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fef10013510 tx=0x7fef1000b040).stop 2023-12-06T03:57:38.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.497+0000 7fef2165a700 1 -- 172.21.15.152:0/3050543192 shutdown_connections 2023-12-06T03:57:38.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.497+0000 7fef2165a700 1 --2- 172.21.15.152:0/3050543192 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fef0402ff60 0x7fef04032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:38.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.497+0000 7fef2165a700 1 --2- 172.21.15.152:0/3050543192 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fef1c0f5410 0x7fef1c105630 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:38.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.497+0000 7fef2165a700 1 -- 172.21.15.152:0/3050543192 >> 172.21.15.152:0/3050543192 conn(0x7fef1c0f0b70 msgr2=0x7fef1c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:38.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.497+0000 7fef2165a700 1 -- 172.21.15.152:0/3050543192 shutdown_connections 2023-12-06T03:57:38.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:38.497+0000 7fef2165a700 1 -- 172.21.15.152:0/3050543192 wait complete. 2023-12-06T03:57:38.499 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:39.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:38 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:38.495888+0000 mon.smithi132 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.152:0/3050543192' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:39.862 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:39.862 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:40.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:39 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:38.428114+0000 mgr.smithi132.gokxja (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:40.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.954+0000 7f539d291700 1 -- 172.21.15.152:0/382288230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f539806b960 msgr2=0x7f5398069e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:40.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.954+0000 7f539d291700 1 --2- 172.21.15.152:0/382288230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f539806b960 0x7f5398069e60 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f5388009fb0 tx=0x7f5388009b30).stop 2023-12-06T03:57:40.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.954+0000 7f539d291700 1 -- 172.21.15.152:0/382288230 shutdown_connections 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.954+0000 7f539d291700 1 --2- 172.21.15.152:0/382288230 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f539806b960 0x7f5398069e60 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.954+0000 7f539d291700 1 -- 172.21.15.152:0/382288230 >> 172.21.15.152:0/382288230 conn(0x7f53980f1b90 msgr2=0x7f53980f3fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.954+0000 7f539d291700 1 -- 172.21.15.152:0/382288230 shutdown_connections 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.954+0000 7f539d291700 1 -- 172.21.15.152:0/382288230 wait complete. 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.955+0000 7f539d291700 1 Processor -- start 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.955+0000 7f539d291700 1 -- start start 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.955+0000 7f539d291700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f539806b960 0x7f5398108840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.955+0000 7f539d291700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5398108d30 con 0x7f539806b960 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.955+0000 7f5396ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f539806b960 0x7f5398108840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.955+0000 7f5396ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f539806b960 0x7f5398108840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:36140/0 (socket says 172.21.15.152:36140) 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.955+0000 7f5396ffd700 1 -- 172.21.15.152:0/1432129014 learned_addr learned my addr 172.21.15.152:0/1432129014 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.956+0000 7f5396ffd700 1 -- 172.21.15.152:0/1432129014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5388005040 con 0x7f539806b960 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.956+0000 7f5396ffd700 1 --2- 172.21.15.152:0/1432129014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f539806b960 0x7f5398108840 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f5388013510 tx=0x7f538800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.956+0000 7f537f7fe700 1 -- 172.21.15.152:0/1432129014 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5388009320 con 0x7f539806b960 2023-12-06T03:57:40.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.956+0000 7f539d291700 1 -- 172.21.15.152:0/1432129014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5398108f30 con 0x7f539806b960 2023-12-06T03:57:40.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.956+0000 7f537f7fe700 1 -- 172.21.15.152:0/1432129014 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5388013f00 con 0x7f539806b960 2023-12-06T03:57:40.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.956+0000 7f537f7fe700 1 -- 172.21.15.152:0/1432129014 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5388014dc0 con 0x7f539806b960 2023-12-06T03:57:40.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.956+0000 7f539d291700 1 -- 172.21.15.152:0/1432129014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53981093c0 con 0x7f539806b960 2023-12-06T03:57:40.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.957+0000 7f537f7fe700 1 -- 172.21.15.152:0/1432129014 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5388014420 con 0x7f539806b960 2023-12-06T03:57:40.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.957+0000 7f539d291700 1 -- 172.21.15.152:0/1432129014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5398045ad0 con 0x7f539806b960 2023-12-06T03:57:40.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.957+0000 7f537f7fe700 1 --2- 172.21.15.152:0/1432129014 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f538002ff60 0x7f5380032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:40.960 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.957+0000 7f537f7fe700 1 -- 172.21.15.152:0/1432129014 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5388041030 con 0x7f539806b960 2023-12-06T03:57:40.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.957+0000 7f53967fc700 1 --2- 172.21.15.152:0/1432129014 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f538002ff60 0x7f5380032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:40.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.958+0000 7f53967fc700 1 --2- 172.21.15.152:0/1432129014 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f538002ff60 0x7f5380032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f538c0096a0 tx=0x7f538c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:40.962 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:40.962+0000 7f537f7fe700 1 -- 172.21.15.152:0/1432129014 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53880095f0 con 0x7f539806b960 2023-12-06T03:57:41.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:41.168+0000 7f539d291700 1 -- 172.21.15.152:0/1432129014 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53980fe0e0 con 0x7f539806b960 2023-12-06T03:57:41.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:41.168+0000 7f537f7fe700 1 -- 172.21.15.152:0/1432129014 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f538801cb10 con 0x7f539806b960 2023-12-06T03:57:41.169 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:41.170 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:41.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:41.170+0000 7f539d291700 1 -- 172.21.15.152:0/1432129014 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f538002ff60 msgr2=0x7f5380032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:41.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:41.170+0000 7f539d291700 1 --2- 172.21.15.152:0/1432129014 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f538002ff60 0x7f5380032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f538c0096a0 tx=0x7f538c006b40).stop 2023-12-06T03:57:41.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:41.170+0000 7f539d291700 1 -- 172.21.15.152:0/1432129014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f539806b960 msgr2=0x7f5398108840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:41.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:41.170+0000 7f539d291700 1 --2- 172.21.15.152:0/1432129014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f539806b960 0x7f5398108840 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f5388013510 tx=0x7f538800b040).stop 2023-12-06T03:57:41.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:41.170+0000 7f539d291700 1 -- 172.21.15.152:0/1432129014 shutdown_connections 2023-12-06T03:57:41.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:41.170+0000 7f539d291700 1 --2- 172.21.15.152:0/1432129014 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f538002ff60 0x7f5380032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:41.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:41.170+0000 7f539d291700 1 --2- 172.21.15.152:0/1432129014 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f539806b960 0x7f5398108840 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:41.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:41.170+0000 7f539d291700 1 -- 172.21.15.152:0/1432129014 >> 172.21.15.152:0/1432129014 conn(0x7f53980f1b90 msgr2=0x7f53980f2640 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:41.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:41.170+0000 7f539d291700 1 -- 172.21.15.152:0/1432129014 shutdown_connections 2023-12-06T03:57:41.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:41.170+0000 7f539d291700 1 -- 172.21.15.152:0/1432129014 wait complete. 2023-12-06T03:57:41.172 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:42.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:41 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:40.428580+0000 mgr.smithi132.gokxja (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:42.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:41 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:41.168986+0000 mon.smithi132 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.152:0/1432129014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:42.502 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:42.502 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:43.626 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.624+0000 7f4603348700 1 -- 172.21.15.152:0/1138882958 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45fc0f6510 msgr2=0x7f45fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:43.626 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.624+0000 7f4603348700 1 --2- 172.21.15.152:0/1138882958 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45fc0f6510 0x7f45fc0f6930 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f45ec009fb0 tx=0x7f45ec009b30).stop 2023-12-06T03:57:43.626 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.624+0000 7f4603348700 1 -- 172.21.15.152:0/1138882958 shutdown_connections 2023-12-06T03:57:43.626 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.624+0000 7f4603348700 1 --2- 172.21.15.152:0/1138882958 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45fc0f6510 0x7f45fc0f6930 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:43.626 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.624+0000 7f4603348700 1 -- 172.21.15.152:0/1138882958 >> 172.21.15.152:0/1138882958 conn(0x7f45fc0f1c70 msgr2=0x7f45fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:43.626 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.624+0000 7f4603348700 1 -- 172.21.15.152:0/1138882958 shutdown_connections 2023-12-06T03:57:43.626 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.624+0000 7f4603348700 1 -- 172.21.15.152:0/1138882958 wait complete. 2023-12-06T03:57:43.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.624+0000 7f4603348700 1 Processor -- start 2023-12-06T03:57:43.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.624+0000 7f4603348700 1 -- start start 2023-12-06T03:57:43.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.624+0000 7f4603348700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45fc0f6510 0x7f45fc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:43.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.624+0000 7f4603348700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45fc106c20 con 0x7f45fc0f6510 2023-12-06T03:57:43.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.625+0000 7f46010e4700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45fc0f6510 0x7f45fc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:43.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.625+0000 7f46010e4700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45fc0f6510 0x7f45fc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:36154/0 (socket says 172.21.15.152:36154) 2023-12-06T03:57:43.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.625+0000 7f46010e4700 1 -- 172.21.15.152:0/1859654854 learned_addr learned my addr 172.21.15.152:0/1859654854 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:43.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.625+0000 7f46010e4700 1 -- 172.21.15.152:0/1859654854 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45ec005040 con 0x7f45fc0f6510 2023-12-06T03:57:43.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.625+0000 7f46010e4700 1 --2- 172.21.15.152:0/1859654854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45fc0f6510 0x7f45fc106730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f45ec013510 tx=0x7f45ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:43.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.625+0000 7f45f1ffb700 1 -- 172.21.15.152:0/1859654854 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45ec009320 con 0x7f45fc0f6510 2023-12-06T03:57:43.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.625+0000 7f45f1ffb700 1 -- 172.21.15.152:0/1859654854 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f45ec013f00 con 0x7f45fc0f6510 2023-12-06T03:57:43.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.625+0000 7f4603348700 1 -- 172.21.15.152:0/1859654854 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45fc106e20 con 0x7f45fc0f6510 2023-12-06T03:57:43.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.625+0000 7f45f1ffb700 1 -- 172.21.15.152:0/1859654854 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f45ec014d90 con 0x7f45fc0f6510 2023-12-06T03:57:43.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.625+0000 7f4603348700 1 -- 172.21.15.152:0/1859654854 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45fc1072b0 con 0x7f45fc0f6510 2023-12-06T03:57:43.628 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.626+0000 7f45f1ffb700 1 -- 172.21.15.152:0/1859654854 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45ec0263a0 con 0x7f45fc0f6510 2023-12-06T03:57:43.628 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.626+0000 7f45f1ffb700 1 --2- 172.21.15.152:0/1859654854 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f45e802ffb0 0x7f45e8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:43.628 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.626+0000 7f4603348700 1 -- 172.21.15.152:0/1859654854 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45fc045ad0 con 0x7f45fc0f6510 2023-12-06T03:57:43.629 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.626+0000 7f45f1ffb700 1 -- 172.21.15.152:0/1859654854 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f45ec0410c0 con 0x7f45fc0f6510 2023-12-06T03:57:43.629 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.626+0000 7f46008e3700 1 --2- 172.21.15.152:0/1859654854 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f45e802ffb0 0x7f45e8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:43.629 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.627+0000 7f46008e3700 1 --2- 172.21.15.152:0/1859654854 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f45e802ffb0 0x7f45e8032470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f45f80096a0 tx=0x7f45f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:43.632 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.631+0000 7f45f1ffb700 1 -- 172.21.15.152:0/1859654854 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45ec012920 con 0x7f45fc0f6510 2023-12-06T03:57:43.838 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.838+0000 7f4603348700 1 -- 172.21.15.152:0/1859654854 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45fc0fc050 con 0x7f45fc0f6510 2023-12-06T03:57:43.839 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.838+0000 7f45f1ffb700 1 -- 172.21.15.152:0/1859654854 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f45ec02b020 con 0x7f45fc0f6510 2023-12-06T03:57:43.839 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:43.839 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:43.841 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.840+0000 7f4603348700 1 -- 172.21.15.152:0/1859654854 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f45e802ffb0 msgr2=0x7f45e8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:43.841 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.840+0000 7f4603348700 1 --2- 172.21.15.152:0/1859654854 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f45e802ffb0 0x7f45e8032470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f45f80096a0 tx=0x7f45f8006b40).stop 2023-12-06T03:57:43.841 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.840+0000 7f4603348700 1 -- 172.21.15.152:0/1859654854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45fc0f6510 msgr2=0x7f45fc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:43.841 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.840+0000 7f4603348700 1 --2- 172.21.15.152:0/1859654854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45fc0f6510 0x7f45fc106730 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f45ec013510 tx=0x7f45ec00b040).stop 2023-12-06T03:57:43.841 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.840+0000 7f4603348700 1 -- 172.21.15.152:0/1859654854 shutdown_connections 2023-12-06T03:57:43.841 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.840+0000 7f4603348700 1 --2- 172.21.15.152:0/1859654854 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f45e802ffb0 0x7f45e8032470 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:43.841 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.840+0000 7f4603348700 1 --2- 172.21.15.152:0/1859654854 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f45fc0f6510 0x7f45fc106730 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:43.842 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.840+0000 7f4603348700 1 -- 172.21.15.152:0/1859654854 >> 172.21.15.152:0/1859654854 conn(0x7f45fc0f1c70 msgr2=0x7f45fc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:43.842 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.840+0000 7f4603348700 1 -- 172.21.15.152:0/1859654854 shutdown_connections 2023-12-06T03:57:43.842 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:43.840+0000 7f4603348700 1 -- 172.21.15.152:0/1859654854 wait complete. 2023-12-06T03:57:43.842 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:44.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:42.428885+0000 mgr.smithi132.gokxja (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:45.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:44 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:43.838763+0000 mon.smithi132 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.152:0/1859654854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:45.209 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:45.209 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:46.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:45 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:44.429265+0000 mgr.smithi132.gokxja (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:46.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.313+0000 7f6663010700 1 -- 172.21.15.152:0/2698003246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f665c060280 msgr2=0x7f665c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:46.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.313+0000 7f6663010700 1 --2- 172.21.15.152:0/2698003246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f665c060280 0x7f665c0f90f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f664c009fb0 tx=0x7f664c009b30).stop 2023-12-06T03:57:46.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.313+0000 7f6663010700 1 -- 172.21.15.152:0/2698003246 shutdown_connections 2023-12-06T03:57:46.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.313+0000 7f6663010700 1 --2- 172.21.15.152:0/2698003246 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f665c060280 0x7f665c0f90f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:46.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.313+0000 7f6663010700 1 -- 172.21.15.152:0/2698003246 >> 172.21.15.152:0/2698003246 conn(0x7f665c0f0b70 msgr2=0x7f665c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:46.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.313+0000 7f6663010700 1 -- 172.21.15.152:0/2698003246 shutdown_connections 2023-12-06T03:57:46.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.313+0000 7f6663010700 1 -- 172.21.15.152:0/2698003246 wait complete. 2023-12-06T03:57:46.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.314+0000 7f6663010700 1 Processor -- start 2023-12-06T03:57:46.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.314+0000 7f6663010700 1 -- start start 2023-12-06T03:57:46.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.314+0000 7f6663010700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f665c060280 0x7f665c1088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:46.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.314+0000 7f6663010700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665c108de0 con 0x7f665c060280 2023-12-06T03:57:46.317 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.314+0000 7f6660dac700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f665c060280 0x7f665c1088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:46.317 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.314+0000 7f6660dac700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f665c060280 0x7f665c1088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:36180/0 (socket says 172.21.15.152:36180) 2023-12-06T03:57:46.317 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.314+0000 7f6660dac700 1 -- 172.21.15.152:0/952287796 learned_addr learned my addr 172.21.15.152:0/952287796 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:46.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.315+0000 7f6660dac700 1 -- 172.21.15.152:0/952287796 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f664c005040 con 0x7f665c060280 2023-12-06T03:57:46.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.315+0000 7f6660dac700 1 --2- 172.21.15.152:0/952287796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f665c060280 0x7f665c1088f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f664c009830 tx=0x7f664c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:46.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.315+0000 7f66597fa700 1 -- 172.21.15.152:0/952287796 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f664c009320 con 0x7f665c060280 2023-12-06T03:57:46.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.315+0000 7f6663010700 1 -- 172.21.15.152:0/952287796 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f665c108fe0 con 0x7f665c060280 2023-12-06T03:57:46.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.315+0000 7f66597fa700 1 -- 172.21.15.152:0/952287796 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f664c0136c0 con 0x7f665c060280 2023-12-06T03:57:46.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.315+0000 7f66597fa700 1 -- 172.21.15.152:0/952287796 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f664c01ee90 con 0x7f665c060280 2023-12-06T03:57:46.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.315+0000 7f6663010700 1 -- 172.21.15.152:0/952287796 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f665c109470 con 0x7f665c060280 2023-12-06T03:57:46.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.316+0000 7f66597fa700 1 -- 172.21.15.152:0/952287796 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f664c014440 con 0x7f665c060280 2023-12-06T03:57:46.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.316+0000 7f66597fa700 1 --2- 172.21.15.152:0/952287796 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f664802ff10 0x7f66480323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:46.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.316+0000 7f66597fa700 1 -- 172.21.15.152:0/952287796 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f664c040fd0 con 0x7f665c060280 2023-12-06T03:57:46.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.316+0000 7f6663010700 1 -- 172.21.15.152:0/952287796 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f665c059100 con 0x7f665c060280 2023-12-06T03:57:46.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.316+0000 7f665bfff700 1 --2- 172.21.15.152:0/952287796 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f664802ff10 0x7f66480323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:46.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.317+0000 7f665bfff700 1 --2- 172.21.15.152:0/952287796 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f664802ff10 0x7f66480323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f66440096a0 tx=0x7f6644006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:46.321 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.321+0000 7f66597fa700 1 -- 172.21.15.152:0/952287796 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f664c0273f0 con 0x7f665c060280 2023-12-06T03:57:46.528 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.527+0000 7f6663010700 1 -- 172.21.15.152:0/952287796 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f665c045ad0 con 0x7f665c060280 2023-12-06T03:57:46.528 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.528+0000 7f66597fa700 1 -- 172.21.15.152:0/952287796 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f664c01c320 con 0x7f665c060280 2023-12-06T03:57:46.529 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:46.529 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:46.531 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.530+0000 7f6663010700 1 -- 172.21.15.152:0/952287796 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f664802ff10 msgr2=0x7f66480323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:46.531 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.530+0000 7f6663010700 1 --2- 172.21.15.152:0/952287796 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f664802ff10 0x7f66480323d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f66440096a0 tx=0x7f6644006b40).stop 2023-12-06T03:57:46.531 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.530+0000 7f6663010700 1 -- 172.21.15.152:0/952287796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f665c060280 msgr2=0x7f665c1088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:46.531 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.530+0000 7f6663010700 1 --2- 172.21.15.152:0/952287796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f665c060280 0x7f665c1088f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f664c009830 tx=0x7f664c00b040).stop 2023-12-06T03:57:46.531 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.530+0000 7f6663010700 1 -- 172.21.15.152:0/952287796 shutdown_connections 2023-12-06T03:57:46.531 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.530+0000 7f6663010700 1 --2- 172.21.15.152:0/952287796 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f664802ff10 0x7f66480323d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:46.531 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.530+0000 7f6663010700 1 --2- 172.21.15.152:0/952287796 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f665c060280 0x7f665c1088f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:46.532 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.530+0000 7f6663010700 1 -- 172.21.15.152:0/952287796 >> 172.21.15.152:0/952287796 conn(0x7f665c0f0b70 msgr2=0x7f665c0f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:46.532 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.530+0000 7f6663010700 1 -- 172.21.15.152:0/952287796 shutdown_connections 2023-12-06T03:57:46.532 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:46.530+0000 7f6663010700 1 -- 172.21.15.152:0/952287796 wait complete. 2023-12-06T03:57:46.532 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:47.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:46 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:46.528309+0000 mon.smithi132 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.152:0/952287796' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:47.903 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:47.903 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:48.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:47 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:46.429699+0000 mgr.smithi132.gokxja (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:49.022 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.019+0000 7f6941005700 1 -- 172.21.15.152:0/1327295972 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f693c0f6510 msgr2=0x7f693c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:49.022 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.019+0000 7f6941005700 1 --2- 172.21.15.152:0/1327295972 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f693c0f6510 0x7f693c0f6930 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f6924009fb0 tx=0x7f6924009b30).stop 2023-12-06T03:57:49.022 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.019+0000 7f6941005700 1 -- 172.21.15.152:0/1327295972 shutdown_connections 2023-12-06T03:57:49.022 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.019+0000 7f6941005700 1 --2- 172.21.15.152:0/1327295972 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f693c0f6510 0x7f693c0f6930 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:49.023 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.019+0000 7f6941005700 1 -- 172.21.15.152:0/1327295972 >> 172.21.15.152:0/1327295972 conn(0x7f693c0f1c70 msgr2=0x7f693c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:49.023 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.019+0000 7f6941005700 1 -- 172.21.15.152:0/1327295972 shutdown_connections 2023-12-06T03:57:49.023 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.019+0000 7f6941005700 1 -- 172.21.15.152:0/1327295972 wait complete. 2023-12-06T03:57:49.023 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.020+0000 7f6941005700 1 Processor -- start 2023-12-06T03:57:49.023 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.020+0000 7f6941005700 1 -- start start 2023-12-06T03:57:49.023 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.020+0000 7f6941005700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f693c0f6510 0x7f693c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:49.023 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.020+0000 7f6941005700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f693c106c20 con 0x7f693c0f6510 2023-12-06T03:57:49.023 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.020+0000 7f693ad9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f693c0f6510 0x7f693c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:49.023 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.020+0000 7f693ad9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f693c0f6510 0x7f693c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:36206/0 (socket says 172.21.15.152:36206) 2023-12-06T03:57:49.023 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.020+0000 7f693ad9d700 1 -- 172.21.15.152:0/290612875 learned_addr learned my addr 172.21.15.152:0/290612875 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:49.023 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.021+0000 7f693ad9d700 1 -- 172.21.15.152:0/290612875 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6924005040 con 0x7f693c0f6510 2023-12-06T03:57:49.024 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.021+0000 7f693ad9d700 1 --2- 172.21.15.152:0/290612875 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f693c0f6510 0x7f693c106730 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f6924013510 tx=0x7f692400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:49.024 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.021+0000 7f69337fe700 1 -- 172.21.15.152:0/290612875 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6924009320 con 0x7f693c0f6510 2023-12-06T03:57:49.024 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.021+0000 7f69337fe700 1 -- 172.21.15.152:0/290612875 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6924013f00 con 0x7f693c0f6510 2023-12-06T03:57:49.024 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.021+0000 7f6941005700 1 -- 172.21.15.152:0/290612875 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f693c106e20 con 0x7f693c0f6510 2023-12-06T03:57:49.024 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.021+0000 7f69337fe700 1 -- 172.21.15.152:0/290612875 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6924014d90 con 0x7f693c0f6510 2023-12-06T03:57:49.024 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.021+0000 7f6941005700 1 -- 172.21.15.152:0/290612875 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f693c1072b0 con 0x7f693c0f6510 2023-12-06T03:57:49.025 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.022+0000 7f69337fe700 1 -- 172.21.15.152:0/290612875 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6924014420 con 0x7f693c0f6510 2023-12-06T03:57:49.025 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.022+0000 7f6941005700 1 -- 172.21.15.152:0/290612875 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f693c045ad0 con 0x7f693c0f6510 2023-12-06T03:57:49.025 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.022+0000 7f69337fe700 1 --2- 172.21.15.152:0/290612875 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f692802ff60 0x7f6928032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:49.026 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.022+0000 7f69337fe700 1 -- 172.21.15.152:0/290612875 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6924041000 con 0x7f693c0f6510 2023-12-06T03:57:49.026 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.022+0000 7f693a59c700 1 --2- 172.21.15.152:0/290612875 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f692802ff60 0x7f6928032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:49.026 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.023+0000 7f693a59c700 1 --2- 172.21.15.152:0/290612875 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f692802ff60 0x7f6928032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f692c0096a0 tx=0x7f692c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:49.028 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.027+0000 7f69337fe700 1 -- 172.21.15.152:0/290612875 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f69240095f0 con 0x7f693c0f6510 2023-12-06T03:57:49.233 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.232+0000 7f6941005700 1 -- 172.21.15.152:0/290612875 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f693c0fbf60 con 0x7f693c0f6510 2023-12-06T03:57:49.234 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.233+0000 7f69337fe700 1 -- 172.21.15.152:0/290612875 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f692401db70 con 0x7f693c0f6510 2023-12-06T03:57:49.234 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:49.234 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:49.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.234+0000 7f6941005700 1 -- 172.21.15.152:0/290612875 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f692802ff60 msgr2=0x7f6928032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:49.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.234+0000 7f6941005700 1 --2- 172.21.15.152:0/290612875 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f692802ff60 0x7f6928032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f692c0096a0 tx=0x7f692c006b40).stop 2023-12-06T03:57:49.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.234+0000 7f6941005700 1 -- 172.21.15.152:0/290612875 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f693c0f6510 msgr2=0x7f693c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:49.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.234+0000 7f6941005700 1 --2- 172.21.15.152:0/290612875 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f693c0f6510 0x7f693c106730 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f6924013510 tx=0x7f692400b040).stop 2023-12-06T03:57:49.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.234+0000 7f6941005700 1 -- 172.21.15.152:0/290612875 shutdown_connections 2023-12-06T03:57:49.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.234+0000 7f6941005700 1 --2- 172.21.15.152:0/290612875 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f692802ff60 0x7f6928032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:49.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.234+0000 7f6941005700 1 --2- 172.21.15.152:0/290612875 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f693c0f6510 0x7f693c106730 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:49.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.234+0000 7f6941005700 1 -- 172.21.15.152:0/290612875 >> 172.21.15.152:0/290612875 conn(0x7f693c0f1c70 msgr2=0x7f693c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:49.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.234+0000 7f6941005700 1 -- 172.21.15.152:0/290612875 shutdown_connections 2023-12-06T03:57:49.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:49.234+0000 7f6941005700 1 -- 172.21.15.152:0/290612875 wait complete. 2023-12-06T03:57:49.236 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:50.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:49 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:48.430070+0000 mgr.smithi132.gokxja (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:50.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:49 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:49.233265+0000 mon.smithi132 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.152:0/290612875' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:50.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:50.597 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:51.741 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.739+0000 7f8a2b293700 1 -- 172.21.15.152:0/780929662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a240f6510 msgr2=0x7f8a240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:51.741 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.739+0000 7f8a2b293700 1 --2- 172.21.15.152:0/780929662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a240f6510 0x7f8a240f6930 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7f8a18009fb0 tx=0x7f8a18009b30).stop 2023-12-06T03:57:51.741 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.739+0000 7f8a2b293700 1 -- 172.21.15.152:0/780929662 shutdown_connections 2023-12-06T03:57:51.741 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.739+0000 7f8a2b293700 1 --2- 172.21.15.152:0/780929662 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a240f6510 0x7f8a240f6930 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:51.741 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.739+0000 7f8a2b293700 1 -- 172.21.15.152:0/780929662 >> 172.21.15.152:0/780929662 conn(0x7f8a240f1c70 msgr2=0x7f8a240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:51.741 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.740+0000 7f8a2b293700 1 -- 172.21.15.152:0/780929662 shutdown_connections 2023-12-06T03:57:51.742 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.740+0000 7f8a2b293700 1 -- 172.21.15.152:0/780929662 wait complete. 2023-12-06T03:57:51.742 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.740+0000 7f8a2b293700 1 Processor -- start 2023-12-06T03:57:51.742 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.740+0000 7f8a2b293700 1 -- start start 2023-12-06T03:57:51.742 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.740+0000 7f8a2b293700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a240f6510 0x7f8a24106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:51.742 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.740+0000 7f8a2b293700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a24106c20 con 0x7f8a240f6510 2023-12-06T03:57:51.744 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.741+0000 7f8a2902f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a240f6510 0x7f8a24106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:51.744 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.741+0000 7f8a2902f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a240f6510 0x7f8a24106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:33052/0 (socket says 172.21.15.152:33052) 2023-12-06T03:57:51.744 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.741+0000 7f8a2902f700 1 -- 172.21.15.152:0/2863747750 learned_addr learned my addr 172.21.15.152:0/2863747750 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:51.744 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.741+0000 7f8a2902f700 1 -- 172.21.15.152:0/2863747750 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a18005040 con 0x7f8a240f6510 2023-12-06T03:57:51.744 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.741+0000 7f8a2902f700 1 --2- 172.21.15.152:0/2863747750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a240f6510 0x7f8a24106730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f8a18013510 tx=0x7f8a1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:51.744 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.741+0000 7f8a15ffb700 1 -- 172.21.15.152:0/2863747750 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a18009320 con 0x7f8a240f6510 2023-12-06T03:57:51.744 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.741+0000 7f8a2b293700 1 -- 172.21.15.152:0/2863747750 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a24106e20 con 0x7f8a240f6510 2023-12-06T03:57:51.745 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.741+0000 7f8a15ffb700 1 -- 172.21.15.152:0/2863747750 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a18013f00 con 0x7f8a240f6510 2023-12-06T03:57:51.745 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.741+0000 7f8a15ffb700 1 -- 172.21.15.152:0/2863747750 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a18014d90 con 0x7f8a240f6510 2023-12-06T03:57:51.745 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.742+0000 7f8a2b293700 1 -- 172.21.15.152:0/2863747750 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a241072b0 con 0x7f8a240f6510 2023-12-06T03:57:51.745 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.742+0000 7f8a15ffb700 1 -- 172.21.15.152:0/2863747750 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a180263a0 con 0x7f8a240f6510 2023-12-06T03:57:51.745 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.743+0000 7f8a15ffb700 1 --2- 172.21.15.152:0/2863747750 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8a1002ff60 0x7f8a10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:51.745 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.743+0000 7f8a15ffb700 1 -- 172.21.15.152:0/2863747750 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a180410c0 con 0x7f8a240f6510 2023-12-06T03:57:51.745 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.743+0000 7f8a2b293700 1 -- 172.21.15.152:0/2863747750 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a24045ad0 con 0x7f8a240f6510 2023-12-06T03:57:51.745 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.743+0000 7f8a2882e700 1 --2- 172.21.15.152:0/2863747750 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8a1002ff60 0x7f8a10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:51.745 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.743+0000 7f8a2882e700 1 --2- 172.21.15.152:0/2863747750 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8a1002ff60 0x7f8a10032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f8a200096a0 tx=0x7f8a20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:51.748 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.747+0000 7f8a15ffb700 1 -- 172.21.15.152:0/2863747750 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a18012920 con 0x7f8a240f6510 2023-12-06T03:57:51.956 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.955+0000 7f8a2b293700 1 -- 172.21.15.152:0/2863747750 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a240fc050 con 0x7f8a240f6510 2023-12-06T03:57:51.956 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.956+0000 7f8a15ffb700 1 -- 172.21.15.152:0/2863747750 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8a1800f340 con 0x7f8a240f6510 2023-12-06T03:57:51.957 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:51.957 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:51.958 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.958+0000 7f8a2b293700 1 -- 172.21.15.152:0/2863747750 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8a1002ff60 msgr2=0x7f8a10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:51.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.958+0000 7f8a2b293700 1 --2- 172.21.15.152:0/2863747750 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8a1002ff60 0x7f8a10032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f8a200096a0 tx=0x7f8a20006b40).stop 2023-12-06T03:57:51.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.958+0000 7f8a2b293700 1 -- 172.21.15.152:0/2863747750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a240f6510 msgr2=0x7f8a24106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:51.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.958+0000 7f8a2b293700 1 --2- 172.21.15.152:0/2863747750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a240f6510 0x7f8a24106730 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7f8a18013510 tx=0x7f8a1800b040).stop 2023-12-06T03:57:51.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.958+0000 7f8a2b293700 1 -- 172.21.15.152:0/2863747750 shutdown_connections 2023-12-06T03:57:51.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.958+0000 7f8a2b293700 1 --2- 172.21.15.152:0/2863747750 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8a1002ff60 0x7f8a10032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:51.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.958+0000 7f8a2b293700 1 --2- 172.21.15.152:0/2863747750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8a240f6510 0x7f8a24106730 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:51.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.958+0000 7f8a2b293700 1 -- 172.21.15.152:0/2863747750 >> 172.21.15.152:0/2863747750 conn(0x7f8a240f1c70 msgr2=0x7f8a240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:51.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.958+0000 7f8a2b293700 1 -- 172.21.15.152:0/2863747750 shutdown_connections 2023-12-06T03:57:51.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:51.958+0000 7f8a2b293700 1 -- 172.21.15.152:0/2863747750 wait complete. 2023-12-06T03:57:51.959 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:52.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:51 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:50.430473+0000 mgr.smithi132.gokxja (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:53.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:52 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:51.956503+0000 mon.smithi132 (mon.0) 223 : audit [DBG] from='client.? 172.21.15.152:0/2863747750' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:53.339 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:53.339 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:54.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:53 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:52.430727+0000 mgr.smithi132.gokxja (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:54.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.443+0000 7f84ea826700 1 -- 172.21.15.152:0/3105652041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84e40f6a10 msgr2=0x7f84e40f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:54.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.443+0000 7f84ea826700 1 --2- 172.21.15.152:0/3105652041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84e40f6a10 0x7f84e40f6e30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f84d8009fb0 tx=0x7f84d8009b30).stop 2023-12-06T03:57:54.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.443+0000 7f84ea826700 1 -- 172.21.15.152:0/3105652041 shutdown_connections 2023-12-06T03:57:54.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.443+0000 7f84ea826700 1 --2- 172.21.15.152:0/3105652041 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84e40f6a10 0x7f84e40f6e30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:54.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.443+0000 7f84ea826700 1 -- 172.21.15.152:0/3105652041 >> 172.21.15.152:0/3105652041 conn(0x7f84e40f02e0 msgr2=0x7f84e40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:54.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.443+0000 7f84ea826700 1 -- 172.21.15.152:0/3105652041 shutdown_connections 2023-12-06T03:57:54.444 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.443+0000 7f84ea826700 1 -- 172.21.15.152:0/3105652041 wait complete. 2023-12-06T03:57:54.445 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.443+0000 7f84ea826700 1 Processor -- start 2023-12-06T03:57:54.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.444+0000 7f84ea826700 1 -- start start 2023-12-06T03:57:54.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.444+0000 7f84ea826700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84e41056d0 0x7f84e4105af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:54.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.444+0000 7f84ea826700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84e4105fe0 con 0x7f84e41056d0 2023-12-06T03:57:54.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.444+0000 7f84e3fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84e41056d0 0x7f84e4105af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:54.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.444+0000 7f84e3fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84e41056d0 0x7f84e4105af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:33078/0 (socket says 172.21.15.152:33078) 2023-12-06T03:57:54.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.444+0000 7f84e3fff700 1 -- 172.21.15.152:0/1618792507 learned_addr learned my addr 172.21.15.152:0/1618792507 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:54.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.445+0000 7f84e3fff700 1 -- 172.21.15.152:0/1618792507 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d8005040 con 0x7f84e41056d0 2023-12-06T03:57:54.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.445+0000 7f84e3fff700 1 --2- 172.21.15.152:0/1618792507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84e41056d0 0x7f84e4105af0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f84d8009f80 tx=0x7f84d8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:54.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.445+0000 7f84e0ff9700 1 -- 172.21.15.152:0/1618792507 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84d80136c0 con 0x7f84e41056d0 2023-12-06T03:57:54.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.445+0000 7f84e0ff9700 1 -- 172.21.15.152:0/1618792507 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84d8013e70 con 0x7f84e41056d0 2023-12-06T03:57:54.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.445+0000 7f84ea826700 1 -- 172.21.15.152:0/1618792507 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84e41061e0 con 0x7f84e41056d0 2023-12-06T03:57:54.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.445+0000 7f84e0ff9700 1 -- 172.21.15.152:0/1618792507 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84d8014420 con 0x7f84e41056d0 2023-12-06T03:57:54.446 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.445+0000 7f84ea826700 1 -- 172.21.15.152:0/1618792507 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84e4106d30 con 0x7f84e41056d0 2023-12-06T03:57:54.447 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.446+0000 7f84e0ff9700 1 -- 172.21.15.152:0/1618792507 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84d8025060 con 0x7f84e41056d0 2023-12-06T03:57:54.447 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.446+0000 7f84ea826700 1 -- 172.21.15.152:0/1618792507 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84e4059100 con 0x7f84e41056d0 2023-12-06T03:57:54.447 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.446+0000 7f84e0ff9700 1 --2- 172.21.15.152:0/1618792507 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f84cc02ff60 0x7f84cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:54.448 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.446+0000 7f84e0ff9700 1 -- 172.21.15.152:0/1618792507 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84d8040f60 con 0x7f84e41056d0 2023-12-06T03:57:54.448 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.446+0000 7f84e37fe700 1 --2- 172.21.15.152:0/1618792507 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f84cc02ff60 0x7f84cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:54.448 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.447+0000 7f84e37fe700 1 --2- 172.21.15.152:0/1618792507 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f84cc02ff60 0x7f84cc032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f84d40096a0 tx=0x7f84d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:54.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.451+0000 7f84e0ff9700 1 -- 172.21.15.152:0/1618792507 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84d801cb30 con 0x7f84e41056d0 2023-12-06T03:57:54.658 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.656+0000 7f84ea826700 1 -- 172.21.15.152:0/1618792507 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84e41074f0 con 0x7f84e41056d0 2023-12-06T03:57:54.658 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.657+0000 7f84e0ff9700 1 -- 172.21.15.152:0/1618792507 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f84d801cb30 con 0x7f84e41056d0 2023-12-06T03:57:54.658 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:54.658 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:54.659 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.658+0000 7f84ea826700 1 -- 172.21.15.152:0/1618792507 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f84cc02ff60 msgr2=0x7f84cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:54.659 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.658+0000 7f84ea826700 1 --2- 172.21.15.152:0/1618792507 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f84cc02ff60 0x7f84cc032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f84d40096a0 tx=0x7f84d4006b40).stop 2023-12-06T03:57:54.660 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.658+0000 7f84ea826700 1 -- 172.21.15.152:0/1618792507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84e41056d0 msgr2=0x7f84e4105af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:54.660 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.658+0000 7f84ea826700 1 --2- 172.21.15.152:0/1618792507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84e41056d0 0x7f84e4105af0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f84d8009f80 tx=0x7f84d8009b30).stop 2023-12-06T03:57:54.660 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.659+0000 7f84ea826700 1 -- 172.21.15.152:0/1618792507 shutdown_connections 2023-12-06T03:57:54.660 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.659+0000 7f84ea826700 1 --2- 172.21.15.152:0/1618792507 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f84cc02ff60 0x7f84cc032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:54.660 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.659+0000 7f84ea826700 1 --2- 172.21.15.152:0/1618792507 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84e41056d0 0x7f84e4105af0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:54.660 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.659+0000 7f84ea826700 1 -- 172.21.15.152:0/1618792507 >> 172.21.15.152:0/1618792507 conn(0x7f84e40f02e0 msgr2=0x7f84e40f0df0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:54.660 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.659+0000 7f84ea826700 1 -- 172.21.15.152:0/1618792507 shutdown_connections 2023-12-06T03:57:54.660 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:54.659+0000 7f84ea826700 1 -- 172.21.15.152:0/1618792507 wait complete. 2023-12-06T03:57:54.660 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:55.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:54.520319+0000 mon.smithi132 (mon.0) 224 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/mirror_snapshot_schedule"}]: dispatch 2023-12-06T03:57:55.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:54.528971+0000 mon.smithi132 (mon.0) 225 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/trash_purge_schedule"}]: dispatch 2023-12-06T03:57:55.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:54.657271+0000 mon.smithi132 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.152:0/1618792507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:56.043 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:56.044 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:56.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:54.431129+0000 mgr.smithi132.gokxja (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:57.166 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.163+0000 7f7b4be4a700 1 -- 172.21.15.152:0/1292579228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b440f5410 msgr2=0x7f7b440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:57.169 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.163+0000 7f7b4be4a700 1 --2- 172.21.15.152:0/1292579228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b440f5410 0x7f7b440f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f7b40009fb0 tx=0x7f7b40009b30).stop 2023-12-06T03:57:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.163+0000 7f7b4be4a700 1 -- 172.21.15.152:0/1292579228 shutdown_connections 2023-12-06T03:57:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.163+0000 7f7b4be4a700 1 --2- 172.21.15.152:0/1292579228 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b440f5410 0x7f7b440f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.163+0000 7f7b4be4a700 1 -- 172.21.15.152:0/1292579228 >> 172.21.15.152:0/1292579228 conn(0x7f7b440f0b70 msgr2=0x7f7b440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.163+0000 7f7b4be4a700 1 -- 172.21.15.152:0/1292579228 shutdown_connections 2023-12-06T03:57:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.163+0000 7f7b4be4a700 1 -- 172.21.15.152:0/1292579228 wait complete. 2023-12-06T03:57:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.163+0000 7f7b4be4a700 1 Processor -- start 2023-12-06T03:57:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.163+0000 7f7b4be4a700 1 -- start start 2023-12-06T03:57:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.164+0000 7f7b4be4a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b440f5410 0x7f7b44105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:57.170 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.164+0000 7f7b4be4a700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b44105b20 con 0x7f7b440f5410 2023-12-06T03:57:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.164+0000 7f7b49be6700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b440f5410 0x7f7b44105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.164+0000 7f7b49be6700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b440f5410 0x7f7b44105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:33098/0 (socket says 172.21.15.152:33098) 2023-12-06T03:57:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.164+0000 7f7b49be6700 1 -- 172.21.15.152:0/3129040022 learned_addr learned my addr 172.21.15.152:0/3129040022 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.164+0000 7f7b49be6700 1 -- 172.21.15.152:0/3129040022 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b40005040 con 0x7f7b440f5410 2023-12-06T03:57:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.164+0000 7f7b49be6700 1 --2- 172.21.15.152:0/3129040022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b440f5410 0x7f7b44105630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f7b40013510 tx=0x7f7b4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.164+0000 7f7b3a7fc700 1 -- 172.21.15.152:0/3129040022 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b40009320 con 0x7f7b440f5410 2023-12-06T03:57:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.164+0000 7f7b4be4a700 1 -- 172.21.15.152:0/3129040022 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b44105d20 con 0x7f7b440f5410 2023-12-06T03:57:57.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.164+0000 7f7b3a7fc700 1 -- 172.21.15.152:0/3129040022 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b40013f00 con 0x7f7b440f5410 2023-12-06T03:57:57.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.164+0000 7f7b3a7fc700 1 -- 172.21.15.152:0/3129040022 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b40014d90 con 0x7f7b440f5410 2023-12-06T03:57:57.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.165+0000 7f7b4be4a700 1 -- 172.21.15.152:0/3129040022 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b441061b0 con 0x7f7b440f5410 2023-12-06T03:57:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.165+0000 7f7b3a7fc700 1 -- 172.21.15.152:0/3129040022 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7b400095f0 con 0x7f7b440f5410 2023-12-06T03:57:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.165+0000 7f7b3a7fc700 1 --2- 172.21.15.152:0/3129040022 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7b3002ff60 0x7f7b30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.165+0000 7f7b3a7fc700 1 -- 172.21.15.152:0/3129040022 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7b400410a0 con 0x7f7b440f5410 2023-12-06T03:57:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.166+0000 7f7b493e5700 1 --2- 172.21.15.152:0/3129040022 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7b3002ff60 0x7f7b30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.166+0000 7f7b4be4a700 1 -- 172.21.15.152:0/3129040022 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b44059100 con 0x7f7b440f5410 2023-12-06T03:57:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.166+0000 7f7b493e5700 1 --2- 172.21.15.152:0/3129040022 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7b3002ff60 0x7f7b30032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f7b340096a0 tx=0x7f7b34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:57.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.170+0000 7f7b3a7fc700 1 -- 172.21.15.152:0/3129040022 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7b4001d7f0 con 0x7f7b440f5410 2023-12-06T03:57:57.378 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.376+0000 7f7b4be4a700 1 -- 172.21.15.152:0/3129040022 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b440fae60 con 0x7f7b440f5410 2023-12-06T03:57:57.378 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.377+0000 7f7b3a7fc700 1 -- 172.21.15.152:0/3129040022 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7b4002b020 con 0x7f7b440f5410 2023-12-06T03:57:57.379 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:57:57.379 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:57:57.380 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.378+0000 7f7b4be4a700 1 -- 172.21.15.152:0/3129040022 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7b3002ff60 msgr2=0x7f7b30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:57.380 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.378+0000 7f7b4be4a700 1 --2- 172.21.15.152:0/3129040022 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7b3002ff60 0x7f7b30032420 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f7b340096a0 tx=0x7f7b34006b40).stop 2023-12-06T03:57:57.380 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.378+0000 7f7b4be4a700 1 -- 172.21.15.152:0/3129040022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b440f5410 msgr2=0x7f7b44105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:57.380 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.378+0000 7f7b4be4a700 1 --2- 172.21.15.152:0/3129040022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b440f5410 0x7f7b44105630 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f7b40013510 tx=0x7f7b4000b040).stop 2023-12-06T03:57:57.380 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.378+0000 7f7b4be4a700 1 -- 172.21.15.152:0/3129040022 shutdown_connections 2023-12-06T03:57:57.380 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.378+0000 7f7b4be4a700 1 --2- 172.21.15.152:0/3129040022 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7b3002ff60 0x7f7b30032420 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:57.380 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.378+0000 7f7b4be4a700 1 --2- 172.21.15.152:0/3129040022 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7b440f5410 0x7f7b44105630 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:57.381 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.378+0000 7f7b4be4a700 1 -- 172.21.15.152:0/3129040022 >> 172.21.15.152:0/3129040022 conn(0x7f7b440f0b70 msgr2=0x7f7b440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:57.381 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.378+0000 7f7b4be4a700 1 -- 172.21.15.152:0/3129040022 shutdown_connections 2023-12-06T03:57:57.381 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:57.378+0000 7f7b4be4a700 1 -- 172.21.15.152:0/3129040022 wait complete. 2023-12-06T03:57:57.381 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:57:58.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:56.431537+0000 mgr.smithi132.gokxja (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:57:58.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:57:57.378173+0000 mon.smithi132 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.152:0/3129040022' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:57:58.754 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:57:58.754 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:57:59.861 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.858+0000 7f9c20fd3700 1 -- 172.21.15.152:0/3025504615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c1c0f5950 msgr2=0x7f9c1c0f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:57:59.861 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.858+0000 7f9c20fd3700 1 --2- 172.21.15.152:0/3025504615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c1c0f5950 0x7f9c1c0f5d70 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f9c10009fb0 tx=0x7f9c10009b30).stop 2023-12-06T03:57:59.861 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.858+0000 7f9c20fd3700 1 -- 172.21.15.152:0/3025504615 shutdown_connections 2023-12-06T03:57:59.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.858+0000 7f9c20fd3700 1 --2- 172.21.15.152:0/3025504615 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c1c0f5950 0x7f9c1c0f5d70 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:57:59.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.858+0000 7f9c20fd3700 1 -- 172.21.15.152:0/3025504615 >> 172.21.15.152:0/3025504615 conn(0x7f9c1c0f0260 msgr2=0x7f9c1c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:57:59.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.858+0000 7f9c20fd3700 1 -- 172.21.15.152:0/3025504615 shutdown_connections 2023-12-06T03:57:59.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.858+0000 7f9c20fd3700 1 -- 172.21.15.152:0/3025504615 wait complete. 2023-12-06T03:57:59.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.859+0000 7f9c20fd3700 1 Processor -- start 2023-12-06T03:57:59.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.859+0000 7f9c20fd3700 1 -- start start 2023-12-06T03:57:59.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.859+0000 7f9c20fd3700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c1c0f5950 0x7f9c1c106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:59.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.859+0000 7f9c20fd3700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c1c106c00 con 0x7f9c1c0f5950 2023-12-06T03:57:59.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.859+0000 7f9c1a59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c1c0f5950 0x7f9c1c106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:59.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.859+0000 7f9c1a59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c1c0f5950 0x7f9c1c106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:60136/0 (socket says 172.21.15.152:60136) 2023-12-06T03:57:59.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.859+0000 7f9c1a59c700 1 -- 172.21.15.152:0/2475659948 learned_addr learned my addr 172.21.15.152:0/2475659948 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:57:59.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.859+0000 7f9c1a59c700 1 -- 172.21.15.152:0/2475659948 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c10005040 con 0x7f9c1c0f5950 2023-12-06T03:57:59.862 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.859+0000 7f9c1a59c700 1 --2- 172.21.15.152:0/2475659948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c1c0f5950 0x7f9c1c106710 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f9c10013510 tx=0x7f9c1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:59.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.860+0000 7f9c0affd700 1 -- 172.21.15.152:0/2475659948 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c10009320 con 0x7f9c1c0f5950 2023-12-06T03:57:59.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.860+0000 7f9c20fd3700 1 -- 172.21.15.152:0/2475659948 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c1c106e00 con 0x7f9c1c0f5950 2023-12-06T03:57:59.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.860+0000 7f9c0affd700 1 -- 172.21.15.152:0/2475659948 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9c10013f00 con 0x7f9c1c0f5950 2023-12-06T03:57:59.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.860+0000 7f9c0affd700 1 -- 172.21.15.152:0/2475659948 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c10014d90 con 0x7f9c1c0f5950 2023-12-06T03:57:59.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.860+0000 7f9c20fd3700 1 -- 172.21.15.152:0/2475659948 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c1c107290 con 0x7f9c1c0f5950 2023-12-06T03:57:59.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.860+0000 7f9c0affd700 1 -- 172.21.15.152:0/2475659948 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9c100095f0 con 0x7f9c1c0f5950 2023-12-06T03:57:59.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.860+0000 7f9c20fd3700 1 -- 172.21.15.152:0/2475659948 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1c045ad0 con 0x7f9c1c0f5950 2023-12-06T03:57:59.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.861+0000 7f9c0affd700 1 --2- 172.21.15.152:0/2475659948 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9c0402ff60 0x7f9c04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:57:59.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.861+0000 7f9c0affd700 1 -- 172.21.15.152:0/2475659948 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9c10041090 con 0x7f9c1c0f5950 2023-12-06T03:57:59.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.861+0000 7f9c19d9b700 1 --2- 172.21.15.152:0/2475659948 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9c0402ff60 0x7f9c04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:57:59.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.861+0000 7f9c19d9b700 1 --2- 172.21.15.152:0/2475659948 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9c0402ff60 0x7f9c04032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f9c0c0096a0 tx=0x7f9c0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:57:59.867 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:57:59.865+0000 7f9c0affd700 1 -- 172.21.15.152:0/2475659948 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9c10014420 con 0x7f9c1c0f5950 2023-12-06T03:58:00.071 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:00.069+0000 7f9c20fd3700 1 -- 172.21.15.152:0/2475659948 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9c1c108140 con 0x7f9c1c0f5950 2023-12-06T03:58:00.071 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:00.070+0000 7f9c0affd700 1 -- 172.21.15.152:0/2475659948 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9c1001d7f0 con 0x7f9c1c0f5950 2023-12-06T03:58:00.072 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:00.072 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:00.073 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:00.071+0000 7f9c20fd3700 1 -- 172.21.15.152:0/2475659948 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9c0402ff60 msgr2=0x7f9c04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:00.073 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:00.071+0000 7f9c20fd3700 1 --2- 172.21.15.152:0/2475659948 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9c0402ff60 0x7f9c04032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f9c0c0096a0 tx=0x7f9c0c006b40).stop 2023-12-06T03:58:00.073 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:00.071+0000 7f9c20fd3700 1 -- 172.21.15.152:0/2475659948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c1c0f5950 msgr2=0x7f9c1c106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:00.074 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:00.071+0000 7f9c20fd3700 1 --2- 172.21.15.152:0/2475659948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c1c0f5950 0x7f9c1c106710 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f9c10013510 tx=0x7f9c1000b040).stop 2023-12-06T03:58:00.074 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:00.071+0000 7f9c20fd3700 1 -- 172.21.15.152:0/2475659948 shutdown_connections 2023-12-06T03:58:00.074 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:00.071+0000 7f9c20fd3700 1 --2- 172.21.15.152:0/2475659948 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9c0402ff60 0x7f9c04032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:00.074 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:00.071+0000 7f9c20fd3700 1 --2- 172.21.15.152:0/2475659948 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9c1c0f5950 0x7f9c1c106710 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:00.074 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:00.071+0000 7f9c20fd3700 1 -- 172.21.15.152:0/2475659948 >> 172.21.15.152:0/2475659948 conn(0x7f9c1c0f0260 msgr2=0x7f9c1c0f0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:00.074 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:00.071+0000 7f9c20fd3700 1 -- 172.21.15.152:0/2475659948 shutdown_connections 2023-12-06T03:58:00.074 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:00.071+0000 7f9c20fd3700 1 -- 172.21.15.152:0/2475659948 wait complete. 2023-12-06T03:58:00.075 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:00.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:57:59 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:57:58.431974+0000 mgr.smithi132.gokxja (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:01.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:00 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:00.071154+0000 mon.smithi132 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.152:0/2475659948' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:01.478 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:01.478 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:02.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:01 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:00.432303+0000 mgr.smithi132.gokxja (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:02.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.600+0000 7fe316e8a700 1 -- 172.21.15.152:0/1807520447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe310060140 msgr2=0x7fe3100f8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:02.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.600+0000 7fe316e8a700 1 --2- 172.21.15.152:0/1807520447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe310060140 0x7fe3100f8fc0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fe30c009fb0 tx=0x7fe30c009b30).stop 2023-12-06T03:58:02.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.600+0000 7fe316e8a700 1 -- 172.21.15.152:0/1807520447 shutdown_connections 2023-12-06T03:58:02.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.600+0000 7fe316e8a700 1 --2- 172.21.15.152:0/1807520447 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe310060140 0x7fe3100f8fc0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:02.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.600+0000 7fe316e8a700 1 -- 172.21.15.152:0/1807520447 >> 172.21.15.152:0/1807520447 conn(0x7fe3100f09a0 msgr2=0x7fe3100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:02.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.600+0000 7fe316e8a700 1 -- 172.21.15.152:0/1807520447 shutdown_connections 2023-12-06T03:58:02.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.600+0000 7fe316e8a700 1 -- 172.21.15.152:0/1807520447 wait complete. 2023-12-06T03:58:02.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.600+0000 7fe316e8a700 1 Processor -- start 2023-12-06T03:58:02.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.600+0000 7fe316e8a700 1 -- start start 2023-12-06T03:58:02.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.600+0000 7fe316e8a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe310060140 0x7fe310106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:02.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.600+0000 7fe316e8a700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe310106a10 con 0x7fe310060140 2023-12-06T03:58:02.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.601+0000 7fe315e88700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe310060140 0x7fe310106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:02.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.601+0000 7fe315e88700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe310060140 0x7fe310106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:60164/0 (socket says 172.21.15.152:60164) 2023-12-06T03:58:02.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.601+0000 7fe315e88700 1 -- 172.21.15.152:0/2520055437 learned_addr learned my addr 172.21.15.152:0/2520055437 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:02.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.601+0000 7fe315e88700 1 -- 172.21.15.152:0/2520055437 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe30c005040 con 0x7fe310060140 2023-12-06T03:58:02.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.601+0000 7fe315e88700 1 --2- 172.21.15.152:0/2520055437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe310060140 0x7fe310106520 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fe30c013510 tx=0x7fe30c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:02.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.601+0000 7fe3067fc700 1 -- 172.21.15.152:0/2520055437 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe30c009320 con 0x7fe310060140 2023-12-06T03:58:02.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.601+0000 7fe3067fc700 1 -- 172.21.15.152:0/2520055437 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe30c013f00 con 0x7fe310060140 2023-12-06T03:58:02.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.601+0000 7fe316e8a700 1 -- 172.21.15.152:0/2520055437 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe310106c10 con 0x7fe310060140 2023-12-06T03:58:02.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.601+0000 7fe3067fc700 1 -- 172.21.15.152:0/2520055437 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe30c014d90 con 0x7fe310060140 2023-12-06T03:58:02.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.601+0000 7fe316e8a700 1 -- 172.21.15.152:0/2520055437 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3101070a0 con 0x7fe310060140 2023-12-06T03:58:02.605 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.602+0000 7fe3067fc700 1 -- 172.21.15.152:0/2520055437 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe30c014420 con 0x7fe310060140 2023-12-06T03:58:02.605 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.602+0000 7fe3067fc700 1 --2- 172.21.15.152:0/2520055437 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe2fc02ff60 0x7fe2fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:02.605 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.602+0000 7fe316e8a700 1 -- 172.21.15.152:0/2520055437 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe310045ad0 con 0x7fe310060140 2023-12-06T03:58:02.606 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.602+0000 7fe3067fc700 1 -- 172.21.15.152:0/2520055437 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe30c041080 con 0x7fe310060140 2023-12-06T03:58:02.606 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.602+0000 7fe315687700 1 --2- 172.21.15.152:0/2520055437 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe2fc02ff60 0x7fe2fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:02.606 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.603+0000 7fe315687700 1 --2- 172.21.15.152:0/2520055437 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe2fc02ff60 0x7fe2fc032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe3000096a0 tx=0x7fe300006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:02.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.607+0000 7fe3067fc700 1 -- 172.21.15.152:0/2520055437 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe30c0095f0 con 0x7fe310060140 2023-12-06T03:58:02.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.814+0000 7fe316e8a700 1 -- 172.21.15.152:0/2520055437 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe310107f50 con 0x7fe310060140 2023-12-06T03:58:02.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.815+0000 7fe3067fc700 1 -- 172.21.15.152:0/2520055437 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe30c01db70 con 0x7fe310060140 2023-12-06T03:58:02.817 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:02.818 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:02.819 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.817+0000 7fe316e8a700 1 -- 172.21.15.152:0/2520055437 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe2fc02ff60 msgr2=0x7fe2fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:02.819 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.817+0000 7fe316e8a700 1 --2- 172.21.15.152:0/2520055437 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe2fc02ff60 0x7fe2fc032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fe3000096a0 tx=0x7fe300006b40).stop 2023-12-06T03:58:02.819 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.817+0000 7fe316e8a700 1 -- 172.21.15.152:0/2520055437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe310060140 msgr2=0x7fe310106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:02.819 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.817+0000 7fe316e8a700 1 --2- 172.21.15.152:0/2520055437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe310060140 0x7fe310106520 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fe30c013510 tx=0x7fe30c00b040).stop 2023-12-06T03:58:02.819 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.817+0000 7fe316e8a700 1 -- 172.21.15.152:0/2520055437 shutdown_connections 2023-12-06T03:58:02.820 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.817+0000 7fe316e8a700 1 --2- 172.21.15.152:0/2520055437 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe2fc02ff60 0x7fe2fc032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:02.820 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.817+0000 7fe316e8a700 1 --2- 172.21.15.152:0/2520055437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe310060140 0x7fe310106520 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:02.820 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.817+0000 7fe316e8a700 1 -- 172.21.15.152:0/2520055437 >> 172.21.15.152:0/2520055437 conn(0x7fe3100f09a0 msgr2=0x7fe3100f14d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:02.820 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.817+0000 7fe316e8a700 1 -- 172.21.15.152:0/2520055437 shutdown_connections 2023-12-06T03:58:02.820 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:02.817+0000 7fe316e8a700 1 -- 172.21.15.152:0/2520055437 wait complete. 2023-12-06T03:58:02.820 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:02.832 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:58:01.850121+0000 mgr.smithi132.gokxja (mgr.14162) 155 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi132 ... 2023-12-06T03:58:02.833 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-12-06T03:58:02.833 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-06T03:58:02.833 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr time="2023-12-06T03:53:40Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-06T03:58:02.833 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr time="2023-12-06T03:54:18Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-06T03:58:02.833 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr time="2023-12-06T03:54:55Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-06T03:58:02.833 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-06T03:58:02.833 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-12-06T03:58:02.833 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-06T03:58:02.833 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr time="2023-12-06T03:56:09Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-06T03:58:02.833 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr time="2023-12-06T03:56:46Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-06T03:58:02.833 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr time="2023-12-06T03:57:24Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-06T03:58:02.833 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-06T03:58:02.834 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Traceback (most recent call last): 2023-12-06T03:58:02.834 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-06T03:58:02.834 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: main() 2023-12-06T03:58:02.834 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-06T03:58:02.834 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: r = ctx.func(ctx) 2023-12-06T03:58:02.834 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-06T03:58:02.834 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: return func(ctx) 2023-12-06T03:58:02.834 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-06T03:58:02.834 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-06T03:58:02.834 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-12-06T03:58:02.834 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-12-06T03:58:02.834 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-06T03:58:02.834 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: raise RuntimeError('uid/gid not found') 2023-12-06T03:58:02.834 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: RuntimeError: uid/gid not found 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Traceback (most recent call last): 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: yield (conn, connr) 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: code, '\n'.join(err))) 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi132 ... 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr time="2023-12-06T03:53:40Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr time="2023-12-06T03:54:18Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr time="2023-12-06T03:54:55Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi132 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-06T03:58:02.835 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr time="2023-12-06T03:56:09Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-06T03:58:02.836 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr time="2023-12-06T03:56:46Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-06T03:58:02.836 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr time="2023-12-06T03:57:24Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-06T03:58:02.836 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-06T03:58:02.836 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Traceback (most recent call last): 2023-12-06T03:58:02.836 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-06T03:58:02.836 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: main() 2023-12-06T03:58:02.836 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-06T03:58:02.836 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: r = ctx.func(ctx) 2023-12-06T03:58:02.836 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-06T03:58:02.836 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: return func(ctx) 2023-12-06T03:58:02.836 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-12-06T03:58:02.836 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-06T03:58:02.836 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-12-06T03:58:02.836 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-12-06T03:58:02.837 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-12-06T03:58:02.837 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: raise RuntimeError('uid/gid not found') 2023-12-06T03:58:02.837 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: RuntimeError: uid/gid not found 2023-12-06T03:58:02.837 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:01.851451+0000 mon.smithi132 (mon.0) 229 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi132", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-06T03:58:02.837 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:01.856553+0000 mon.smithi132 (mon.0) 230 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi132", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-06T03:58:02.837 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:01.857287+0000 mon.smithi132 (mon.0) 231 : audit [DBG] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-06T03:58:02.837 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:58:01.857960+0000 mgr.smithi132.gokxja (mgr.14162) 156 : cephadm [INF] Deploying daemon crash.smithi132 on smithi132 2023-12-06T03:58:04.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:02.432571+0000 mgr.smithi132.gokxja (mgr.14162) 157 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:04.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:02.816927+0000 mon.smithi132 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.152:0/2520055437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:04.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:03.309725+0000 mon.smithi132 (mon.0) 233 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' 2023-12-06T03:58:04.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:03.483989+0000 mon.smithi132 (mon.0) 234 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' 2023-12-06T03:58:04.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:03.486334+0000 mon.smithi132 (mon.0) 235 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' 2023-12-06T03:58:04.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:03.486695+0000 mon.smithi132 (mon.0) 236 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-06T03:58:04.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:03.489485+0000 mon.smithi132 (mon.0) 237 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' 2023-12-06T03:58:04.200 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:04.201 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:05.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:04 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:03.487138+0000 mgr.smithi132.gokxja (mgr.14162) 158 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-06T03:58:05.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:04 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:58:03.492762+0000 mgr.smithi132.gokxja (mgr.14162) 159 : cephadm [INF] Deploying daemon grafana.smithi132 on smithi132 2023-12-06T03:58:05.339 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.336+0000 7fb9afe7b700 1 -- 172.21.15.152:0/228916341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9a80f4740 msgr2=0x7fb9a80f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:05.339 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.336+0000 7fb9afe7b700 1 --2- 172.21.15.152:0/228916341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9a80f4740 0x7fb9a80f4b60 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fb99c009fb0 tx=0x7fb99c009b30).stop 2023-12-06T03:58:05.339 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.336+0000 7fb9afe7b700 1 -- 172.21.15.152:0/228916341 shutdown_connections 2023-12-06T03:58:05.339 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.336+0000 7fb9afe7b700 1 --2- 172.21.15.152:0/228916341 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9a80f4740 0x7fb9a80f4b60 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:05.339 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.336+0000 7fb9afe7b700 1 -- 172.21.15.152:0/228916341 >> 172.21.15.152:0/228916341 conn(0x7fb9a80f0260 msgr2=0x7fb9a80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:05.339 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.336+0000 7fb9afe7b700 1 -- 172.21.15.152:0/228916341 shutdown_connections 2023-12-06T03:58:05.339 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.336+0000 7fb9afe7b700 1 -- 172.21.15.152:0/228916341 wait complete. 2023-12-06T03:58:05.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.336+0000 7fb9afe7b700 1 Processor -- start 2023-12-06T03:58:05.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.336+0000 7fb9afe7b700 1 -- start start 2023-12-06T03:58:05.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.336+0000 7fb9afe7b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9a80f4740 0x7fb9a81066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:05.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.336+0000 7fb9afe7b700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9a8106bc0 con 0x7fb9a80f4740 2023-12-06T03:58:05.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.337+0000 7fb9adc17700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9a80f4740 0x7fb9a81066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:05.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.337+0000 7fb9adc17700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9a80f4740 0x7fb9a81066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:60182/0 (socket says 172.21.15.152:60182) 2023-12-06T03:58:05.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.337+0000 7fb9adc17700 1 -- 172.21.15.152:0/2283593665 learned_addr learned my addr 172.21.15.152:0/2283593665 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:05.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.337+0000 7fb9adc17700 1 -- 172.21.15.152:0/2283593665 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb99c005040 con 0x7fb9a80f4740 2023-12-06T03:58:05.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.337+0000 7fb9adc17700 1 --2- 172.21.15.152:0/2283593665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9a80f4740 0x7fb9a81066d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fb99c013510 tx=0x7fb99c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:05.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.337+0000 7fb99a7fc700 1 -- 172.21.15.152:0/2283593665 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb99c009320 con 0x7fb9a80f4740 2023-12-06T03:58:05.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.337+0000 7fb9afe7b700 1 -- 172.21.15.152:0/2283593665 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9a8106dc0 con 0x7fb9a80f4740 2023-12-06T03:58:05.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.337+0000 7fb99a7fc700 1 -- 172.21.15.152:0/2283593665 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb99c013f00 con 0x7fb9a80f4740 2023-12-06T03:58:05.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.337+0000 7fb99a7fc700 1 -- 172.21.15.152:0/2283593665 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb99c014d90 con 0x7fb9a80f4740 2023-12-06T03:58:05.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.337+0000 7fb9afe7b700 1 -- 172.21.15.152:0/2283593665 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9a8107250 con 0x7fb9a80f4740 2023-12-06T03:58:05.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.338+0000 7fb99a7fc700 1 -- 172.21.15.152:0/2283593665 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb99c0263a0 con 0x7fb9a80f4740 2023-12-06T03:58:05.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.338+0000 7fb99a7fc700 1 --2- 172.21.15.152:0/2283593665 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb99402ffb0 0x7fb994032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:05.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.338+0000 7fb99a7fc700 1 -- 172.21.15.152:0/2283593665 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb99c0413c0 con 0x7fb9a80f4740 2023-12-06T03:58:05.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.338+0000 7fb9afe7b700 1 -- 172.21.15.152:0/2283593665 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9a8045ad0 con 0x7fb9a80f4740 2023-12-06T03:58:05.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.338+0000 7fb9ad416700 1 --2- 172.21.15.152:0/2283593665 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb99402ffb0 0x7fb994032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:05.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.339+0000 7fb9ad416700 1 --2- 172.21.15.152:0/2283593665 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb99402ffb0 0x7fb994032470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fb9a40096a0 tx=0x7fb9a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:05.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.343+0000 7fb99a7fc700 1 -- 172.21.15.152:0/2283593665 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb99c012920 con 0x7fb9a80f4740 2023-12-06T03:58:05.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.550+0000 7fb9afe7b700 1 -- 172.21.15.152:0/2283593665 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9a8108010 con 0x7fb9a80f4740 2023-12-06T03:58:05.553 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.551+0000 7fb99a7fc700 1 -- 172.21.15.152:0/2283593665 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb99c02b030 con 0x7fb9a80f4740 2023-12-06T03:58:05.553 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:05.553 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:05.555 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.553+0000 7fb9afe7b700 1 -- 172.21.15.152:0/2283593665 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb99402ffb0 msgr2=0x7fb994032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:05.555 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.553+0000 7fb9afe7b700 1 --2- 172.21.15.152:0/2283593665 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb99402ffb0 0x7fb994032470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fb9a40096a0 tx=0x7fb9a4006b40).stop 2023-12-06T03:58:05.555 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.553+0000 7fb9afe7b700 1 -- 172.21.15.152:0/2283593665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9a80f4740 msgr2=0x7fb9a81066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:05.555 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.553+0000 7fb9afe7b700 1 --2- 172.21.15.152:0/2283593665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9a80f4740 0x7fb9a81066d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fb99c013510 tx=0x7fb99c00b040).stop 2023-12-06T03:58:05.555 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.553+0000 7fb9afe7b700 1 -- 172.21.15.152:0/2283593665 shutdown_connections 2023-12-06T03:58:05.555 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.553+0000 7fb9afe7b700 1 --2- 172.21.15.152:0/2283593665 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb99402ffb0 0x7fb994032470 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:05.555 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.553+0000 7fb9afe7b700 1 --2- 172.21.15.152:0/2283593665 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9a80f4740 0x7fb9a81066d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:05.555 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.553+0000 7fb9afe7b700 1 -- 172.21.15.152:0/2283593665 >> 172.21.15.152:0/2283593665 conn(0x7fb9a80f0260 msgr2=0x7fb9a80f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:05.555 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.553+0000 7fb9afe7b700 1 -- 172.21.15.152:0/2283593665 shutdown_connections 2023-12-06T03:58:05.555 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:05.553+0000 7fb9afe7b700 1 -- 172.21.15.152:0/2283593665 wait complete. 2023-12-06T03:58:05.555 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:06.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:04.432791+0000 mgr.smithi132.gokxja (mgr.14162) 160 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:06.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:05.552663+0000 mon.smithi132 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.152:0/2283593665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:06.916 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:06.916 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:08.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:07 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:06.433110+0000 mgr.smithi132.gokxja (mgr.14162) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:08.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.127+0000 7f8233e35700 1 -- 172.21.15.152:0/2067692634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f822c060280 msgr2=0x7f822c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:08.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.127+0000 7f8233e35700 1 --2- 172.21.15.152:0/2067692634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f822c060280 0x7f822c0f90f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f8220009fb0 tx=0x7f8220009b30).stop 2023-12-06T03:58:08.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.127+0000 7f8233e35700 1 -- 172.21.15.152:0/2067692634 shutdown_connections 2023-12-06T03:58:08.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.127+0000 7f8233e35700 1 --2- 172.21.15.152:0/2067692634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f822c060280 0x7f822c0f90f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:08.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.127+0000 7f8233e35700 1 -- 172.21.15.152:0/2067692634 >> 172.21.15.152:0/2067692634 conn(0x7f822c0f0b70 msgr2=0x7f822c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:08.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.127+0000 7f8233e35700 1 -- 172.21.15.152:0/2067692634 shutdown_connections 2023-12-06T03:58:08.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.127+0000 7f8233e35700 1 -- 172.21.15.152:0/2067692634 wait complete. 2023-12-06T03:58:08.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.128+0000 7f8233e35700 1 Processor -- start 2023-12-06T03:58:08.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.128+0000 7f8233e35700 1 -- start start 2023-12-06T03:58:08.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.128+0000 7f8233e35700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f822c060280 0x7f822c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:08.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.128+0000 7f8233e35700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f822c106bd0 con 0x7f822c060280 2023-12-06T03:58:08.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.129+0000 7f8231bd1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f822c060280 0x7f822c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:08.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.129+0000 7f8231bd1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f822c060280 0x7f822c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:60206/0 (socket says 172.21.15.152:60206) 2023-12-06T03:58:08.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.129+0000 7f8231bd1700 1 -- 172.21.15.152:0/2867733353 learned_addr learned my addr 172.21.15.152:0/2867733353 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:08.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.129+0000 7f8231bd1700 1 -- 172.21.15.152:0/2867733353 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8220005040 con 0x7f822c060280 2023-12-06T03:58:08.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.129+0000 7f8231bd1700 1 --2- 172.21.15.152:0/2867733353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f822c060280 0x7f822c1066e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f8220013510 tx=0x7f822000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:08.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.129+0000 7f821e7fc700 1 -- 172.21.15.152:0/2867733353 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8220009320 con 0x7f822c060280 2023-12-06T03:58:08.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.129+0000 7f821e7fc700 1 -- 172.21.15.152:0/2867733353 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8220013f00 con 0x7f822c060280 2023-12-06T03:58:08.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.129+0000 7f8233e35700 1 -- 172.21.15.152:0/2867733353 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f822c106dd0 con 0x7f822c060280 2023-12-06T03:58:08.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.129+0000 7f821e7fc700 1 -- 172.21.15.152:0/2867733353 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8220014dc0 con 0x7f822c060280 2023-12-06T03:58:08.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.129+0000 7f8233e35700 1 -- 172.21.15.152:0/2867733353 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f822c107260 con 0x7f822c060280 2023-12-06T03:58:08.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.130+0000 7f821e7fc700 1 -- 172.21.15.152:0/2867733353 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8220014420 con 0x7f822c060280 2023-12-06T03:58:08.135 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.130+0000 7f8233e35700 1 -- 172.21.15.152:0/2867733353 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f822c045ad0 con 0x7f822c060280 2023-12-06T03:58:08.135 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.130+0000 7f821e7fc700 1 --2- 172.21.15.152:0/2867733353 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f821802ff60 0x7f8218032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:08.135 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.130+0000 7f821e7fc700 1 -- 172.21.15.152:0/2867733353 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f82200410f0 con 0x7f822c060280 2023-12-06T03:58:08.135 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.130+0000 7f82313d0700 1 --2- 172.21.15.152:0/2867733353 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f821802ff60 0x7f8218032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:08.135 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.131+0000 7f82313d0700 1 --2- 172.21.15.152:0/2867733353 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f821802ff60 0x7f8218032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f82280096a0 tx=0x7f8228006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:08.138 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.135+0000 7f821e7fc700 1 -- 172.21.15.152:0/2867733353 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f82200095f0 con 0x7f822c060280 2023-12-06T03:58:08.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.339+0000 7f8233e35700 1 -- 172.21.15.152:0/2867733353 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f822c108110 con 0x7f822c060280 2023-12-06T03:58:08.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.340+0000 7f821e7fc700 1 -- 172.21.15.152:0/2867733353 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f822001cb10 con 0x7f822c060280 2023-12-06T03:58:08.342 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:08.342 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:08.343 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.341+0000 7f8233e35700 1 -- 172.21.15.152:0/2867733353 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f821802ff60 msgr2=0x7f8218032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:08.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.341+0000 7f8233e35700 1 --2- 172.21.15.152:0/2867733353 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f821802ff60 0x7f8218032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f82280096a0 tx=0x7f8228006b40).stop 2023-12-06T03:58:08.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.341+0000 7f8233e35700 1 -- 172.21.15.152:0/2867733353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f822c060280 msgr2=0x7f822c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:08.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.341+0000 7f8233e35700 1 --2- 172.21.15.152:0/2867733353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f822c060280 0x7f822c1066e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f8220013510 tx=0x7f822000b040).stop 2023-12-06T03:58:08.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.341+0000 7f8233e35700 1 -- 172.21.15.152:0/2867733353 shutdown_connections 2023-12-06T03:58:08.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.341+0000 7f8233e35700 1 --2- 172.21.15.152:0/2867733353 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f821802ff60 0x7f8218032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:08.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.341+0000 7f8233e35700 1 --2- 172.21.15.152:0/2867733353 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f822c060280 0x7f822c1066e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:08.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.341+0000 7f8233e35700 1 -- 172.21.15.152:0/2867733353 >> 172.21.15.152:0/2867733353 conn(0x7f822c0f0b70 msgr2=0x7f822c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:08.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.341+0000 7f8233e35700 1 -- 172.21.15.152:0/2867733353 shutdown_connections 2023-12-06T03:58:08.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:08.341+0000 7f8233e35700 1 -- 172.21.15.152:0/2867733353 wait complete. 2023-12-06T03:58:08.344 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:09.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:08 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:08.341283+0000 mon.smithi132 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.152:0/2867733353' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:09.746 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:09.746 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:10.111 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:09 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:08.433447+0000 mgr.smithi132.gokxja (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:10.811 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.807+0000 7fefc179b700 1 -- 172.21.15.152:0/2667341749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefbc0f6510 msgr2=0x7fefbc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:10.812 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.807+0000 7fefc179b700 1 --2- 172.21.15.152:0/2667341749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefbc0f6510 0x7fefbc0f6930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fefac009fb0 tx=0x7fefac009b30).stop 2023-12-06T03:58:10.812 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.807+0000 7fefc179b700 1 -- 172.21.15.152:0/2667341749 shutdown_connections 2023-12-06T03:58:10.812 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.807+0000 7fefc179b700 1 --2- 172.21.15.152:0/2667341749 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefbc0f6510 0x7fefbc0f6930 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:10.812 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.807+0000 7fefc179b700 1 -- 172.21.15.152:0/2667341749 >> 172.21.15.152:0/2667341749 conn(0x7fefbc0f1c70 msgr2=0x7fefbc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:10.812 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.807+0000 7fefc179b700 1 -- 172.21.15.152:0/2667341749 shutdown_connections 2023-12-06T03:58:10.813 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.807+0000 7fefc179b700 1 -- 172.21.15.152:0/2667341749 wait complete. 2023-12-06T03:58:10.813 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.808+0000 7fefc179b700 1 Processor -- start 2023-12-06T03:58:10.813 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.808+0000 7fefc179b700 1 -- start start 2023-12-06T03:58:10.813 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.808+0000 7fefc179b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefbc0f6510 0x7fefbc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:10.813 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.808+0000 7fefc179b700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefbc069e20 con 0x7fefbc0f6510 2023-12-06T03:58:10.813 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.808+0000 7fefbaffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefbc0f6510 0x7fefbc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:10.813 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.808+0000 7fefbaffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefbc0f6510 0x7fefbc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:35704/0 (socket says 172.21.15.152:35704) 2023-12-06T03:58:10.814 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.808+0000 7fefbaffd700 1 -- 172.21.15.152:0/2889975918 learned_addr learned my addr 172.21.15.152:0/2889975918 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:10.814 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.808+0000 7fefbaffd700 1 -- 172.21.15.152:0/2889975918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefac005040 con 0x7fefbc0f6510 2023-12-06T03:58:10.814 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.808+0000 7fefbaffd700 1 --2- 172.21.15.152:0/2889975918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefbc0f6510 0x7fefbc06b9a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fefac013510 tx=0x7fefac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:10.814 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.808+0000 7fefa37fe700 1 -- 172.21.15.152:0/2889975918 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefac009320 con 0x7fefbc0f6510 2023-12-06T03:58:10.814 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.809+0000 7fefc179b700 1 -- 172.21.15.152:0/2889975918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefbc069fc0 con 0x7fefbc0f6510 2023-12-06T03:58:10.814 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.809+0000 7fefa37fe700 1 -- 172.21.15.152:0/2889975918 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fefac013f00 con 0x7fefbc0f6510 2023-12-06T03:58:10.814 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.809+0000 7fefa37fe700 1 -- 172.21.15.152:0/2889975918 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefac014d90 con 0x7fefbc0f6510 2023-12-06T03:58:10.814 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.809+0000 7fefc179b700 1 -- 172.21.15.152:0/2889975918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefbc06a450 con 0x7fefbc0f6510 2023-12-06T03:58:10.815 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.809+0000 7fefa37fe700 1 -- 172.21.15.152:0/2889975918 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fefac0263a0 con 0x7fefbc0f6510 2023-12-06T03:58:10.815 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.809+0000 7fefc179b700 1 -- 172.21.15.152:0/2889975918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefbc059100 con 0x7fefbc0f6510 2023-12-06T03:58:10.815 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.809+0000 7fefa37fe700 1 --2- 172.21.15.152:0/2889975918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fefa402ff60 0x7fefa4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:10.815 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.809+0000 7fefa37fe700 1 -- 172.21.15.152:0/2889975918 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fefac0413f0 con 0x7fefbc0f6510 2023-12-06T03:58:10.815 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.810+0000 7fefba7fc700 1 --2- 172.21.15.152:0/2889975918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fefa402ff60 0x7fefa4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:10.815 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.810+0000 7fefba7fc700 1 --2- 172.21.15.152:0/2889975918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fefa402ff60 0x7fefa4032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fefb00096a0 tx=0x7fefb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:10.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:10.814+0000 7fefa37fe700 1 -- 172.21.15.152:0/2889975918 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fefac01cb10 con 0x7fefbc0f6510 2023-12-06T03:58:11.022 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:11.020+0000 7fefc179b700 1 -- 172.21.15.152:0/2889975918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefbc0fbf60 con 0x7fefbc0f6510 2023-12-06T03:58:11.023 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:11.021+0000 7fefa37fe700 1 -- 172.21.15.152:0/2889975918 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fefac01cb10 con 0x7fefbc0f6510 2023-12-06T03:58:11.023 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:11.023 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:11.025 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:11.023+0000 7fefc179b700 1 -- 172.21.15.152:0/2889975918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fefa402ff60 msgr2=0x7fefa4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:11.025 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:11.023+0000 7fefc179b700 1 --2- 172.21.15.152:0/2889975918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fefa402ff60 0x7fefa4032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fefb00096a0 tx=0x7fefb0006b40).stop 2023-12-06T03:58:11.025 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:11.023+0000 7fefc179b700 1 -- 172.21.15.152:0/2889975918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefbc0f6510 msgr2=0x7fefbc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:11.025 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:11.023+0000 7fefc179b700 1 --2- 172.21.15.152:0/2889975918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefbc0f6510 0x7fefbc06b9a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fefac013510 tx=0x7fefac00b040).stop 2023-12-06T03:58:11.025 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:11.023+0000 7fefc179b700 1 -- 172.21.15.152:0/2889975918 shutdown_connections 2023-12-06T03:58:11.025 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:11.023+0000 7fefc179b700 1 --2- 172.21.15.152:0/2889975918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fefa402ff60 0x7fefa4032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:11.025 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:11.023+0000 7fefc179b700 1 --2- 172.21.15.152:0/2889975918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fefbc0f6510 0x7fefbc06b9a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:11.025 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:11.023+0000 7fefc179b700 1 -- 172.21.15.152:0/2889975918 >> 172.21.15.152:0/2889975918 conn(0x7fefbc0f1c70 msgr2=0x7fefbc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:11.025 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:11.023+0000 7fefc179b700 1 -- 172.21.15.152:0/2889975918 shutdown_connections 2023-12-06T03:58:11.026 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:11.023+0000 7fefc179b700 1 -- 172.21.15.152:0/2889975918 wait complete. 2023-12-06T03:58:11.026 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:12.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:11 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:10.433757+0000 mgr.smithi132.gokxja (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:12.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:11 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:11.022508+0000 mon.smithi132 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.152:0/2889975918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:12.423 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:12.424 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:13.510 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.509+0000 7f549bfff700 1 -- 172.21.15.152:0/1645579124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c0f51a0 msgr2=0x7f549c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:13.511 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.509+0000 7f549bfff700 1 --2- 172.21.15.152:0/1645579124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c0f51a0 0x7f549c0f55c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f548c009fb0 tx=0x7f548c009b30).stop 2023-12-06T03:58:13.511 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.509+0000 7f549bfff700 1 -- 172.21.15.152:0/1645579124 shutdown_connections 2023-12-06T03:58:13.511 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.509+0000 7f549bfff700 1 --2- 172.21.15.152:0/1645579124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c0f51a0 0x7f549c0f55c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:13.511 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.509+0000 7f549bfff700 1 -- 172.21.15.152:0/1645579124 >> 172.21.15.152:0/1645579124 conn(0x7f549c0f09a0 msgr2=0x7f549c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:13.511 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.509+0000 7f549bfff700 1 -- 172.21.15.152:0/1645579124 shutdown_connections 2023-12-06T03:58:13.511 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.509+0000 7f549bfff700 1 -- 172.21.15.152:0/1645579124 wait complete. 2023-12-06T03:58:13.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.509+0000 7f549bfff700 1 Processor -- start 2023-12-06T03:58:13.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.509+0000 7f549bfff700 1 -- start start 2023-12-06T03:58:13.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.509+0000 7f549bfff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c0f51a0 0x7f549c1091e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:13.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.509+0000 7f549bfff700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f549c060240 con 0x7f549c0f51a0 2023-12-06T03:58:13.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.510+0000 7f549affd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c0f51a0 0x7f549c1091e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:13.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.510+0000 7f549affd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c0f51a0 0x7f549c1091e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:35718/0 (socket says 172.21.15.152:35718) 2023-12-06T03:58:13.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.510+0000 7f549affd700 1 -- 172.21.15.152:0/3503377581 learned_addr learned my addr 172.21.15.152:0/3503377581 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:13.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.510+0000 7f549affd700 1 -- 172.21.15.152:0/3503377581 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f548c005040 con 0x7f549c0f51a0 2023-12-06T03:58:13.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.510+0000 7f549affd700 1 --2- 172.21.15.152:0/3503377581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c0f51a0 0x7f549c1091e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f548c009830 tx=0x7f548c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:13.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.510+0000 7f54837fe700 1 -- 172.21.15.152:0/3503377581 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f548c009320 con 0x7f549c0f51a0 2023-12-06T03:58:13.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.510+0000 7f54837fe700 1 -- 172.21.15.152:0/3503377581 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f548c0136c0 con 0x7f549c0f51a0 2023-12-06T03:58:13.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.510+0000 7f549bfff700 1 -- 172.21.15.152:0/3503377581 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f549c060440 con 0x7f549c0f51a0 2023-12-06T03:58:13.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.510+0000 7f54837fe700 1 -- 172.21.15.152:0/3503377581 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f548c01ee90 con 0x7f549c0f51a0 2023-12-06T03:58:13.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.510+0000 7f549bfff700 1 -- 172.21.15.152:0/3503377581 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f549c109890 con 0x7f549c0f51a0 2023-12-06T03:58:13.514 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.511+0000 7f54837fe700 1 -- 172.21.15.152:0/3503377581 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f548c01e420 con 0x7f549c0f51a0 2023-12-06T03:58:13.514 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.511+0000 7f549bfff700 1 -- 172.21.15.152:0/3503377581 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f549c045ad0 con 0x7f549c0f51a0 2023-12-06T03:58:13.514 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.511+0000 7f54837fe700 1 --2- 172.21.15.152:0/3503377581 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f548402ff10 0x7f54840323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:13.514 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.511+0000 7f54837fe700 1 -- 172.21.15.152:0/3503377581 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f548c041020 con 0x7f549c0f51a0 2023-12-06T03:58:13.514 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.511+0000 7f549a7fc700 1 --2- 172.21.15.152:0/3503377581 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f548402ff10 0x7f54840323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:13.514 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.512+0000 7f549a7fc700 1 --2- 172.21.15.152:0/3503377581 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f548402ff10 0x7f54840323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f54900096a0 tx=0x7f5490006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:13.517 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.516+0000 7f54837fe700 1 -- 172.21.15.152:0/3503377581 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f548c00e2a0 con 0x7f549c0f51a0 2023-12-06T03:58:13.721 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.719+0000 7f549bfff700 1 -- 172.21.15.152:0/3503377581 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f549c10a0b0 con 0x7f549c0f51a0 2023-12-06T03:58:13.721 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.720+0000 7f54837fe700 1 -- 172.21.15.152:0/3503377581 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f548c00e2a0 con 0x7f549c0f51a0 2023-12-06T03:58:13.722 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:13.722 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:13.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.722+0000 7f549bfff700 1 -- 172.21.15.152:0/3503377581 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f548402ff10 msgr2=0x7f54840323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:13.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.722+0000 7f549bfff700 1 --2- 172.21.15.152:0/3503377581 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f548402ff10 0x7f54840323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f54900096a0 tx=0x7f5490006b40).stop 2023-12-06T03:58:13.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.722+0000 7f549bfff700 1 -- 172.21.15.152:0/3503377581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c0f51a0 msgr2=0x7f549c1091e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:13.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.722+0000 7f549bfff700 1 --2- 172.21.15.152:0/3503377581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c0f51a0 0x7f549c1091e0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f548c009830 tx=0x7f548c00b040).stop 2023-12-06T03:58:13.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.722+0000 7f549bfff700 1 -- 172.21.15.152:0/3503377581 shutdown_connections 2023-12-06T03:58:13.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.722+0000 7f549bfff700 1 --2- 172.21.15.152:0/3503377581 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f548402ff10 0x7f54840323d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:13.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.722+0000 7f549bfff700 1 --2- 172.21.15.152:0/3503377581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f549c0f51a0 0x7f549c1091e0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:13.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.722+0000 7f549bfff700 1 -- 172.21.15.152:0/3503377581 >> 172.21.15.152:0/3503377581 conn(0x7f549c0f09a0 msgr2=0x7f549c0f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:13.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.722+0000 7f549bfff700 1 -- 172.21.15.152:0/3503377581 shutdown_connections 2023-12-06T03:58:13.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:13.722+0000 7f549bfff700 1 -- 172.21.15.152:0/3503377581 wait complete. 2023-12-06T03:58:13.725 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:14.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:13 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:12.434048+0000 mgr.smithi132.gokxja (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:14.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:13 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:13.721327+0000 mon.smithi132 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.152:0/3503377581' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:15.081 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:15.304 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:16.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:15 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:14.434441+0000 mgr.smithi132.gokxja (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:16.392 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.389+0000 7fa665dce700 1 -- 172.21.15.152:0/1424179124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6600f6510 msgr2=0x7fa6600f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:16.392 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.389+0000 7fa665dce700 1 --2- 172.21.15.152:0/1424179124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6600f6510 0x7fa6600f6930 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fa654009fb0 tx=0x7fa654009b30).stop 2023-12-06T03:58:16.392 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.389+0000 7fa665dce700 1 -- 172.21.15.152:0/1424179124 shutdown_connections 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.389+0000 7fa665dce700 1 --2- 172.21.15.152:0/1424179124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6600f6510 0x7fa6600f6930 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.389+0000 7fa665dce700 1 -- 172.21.15.152:0/1424179124 >> 172.21.15.152:0/1424179124 conn(0x7fa6600f1c70 msgr2=0x7fa6600f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.389+0000 7fa665dce700 1 -- 172.21.15.152:0/1424179124 shutdown_connections 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.389+0000 7fa665dce700 1 -- 172.21.15.152:0/1424179124 wait complete. 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.389+0000 7fa665dce700 1 Processor -- start 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.390+0000 7fa665dce700 1 -- start start 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.390+0000 7fa665dce700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6600f6510 0x7fa660106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.390+0000 7fa665dce700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa660106c20 con 0x7fa6600f6510 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.390+0000 7fa65f7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6600f6510 0x7fa660106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.390+0000 7fa65f7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6600f6510 0x7fa660106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:35732/0 (socket says 172.21.15.152:35732) 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.390+0000 7fa65f7fe700 1 -- 172.21.15.152:0/2738010887 learned_addr learned my addr 172.21.15.152:0/2738010887 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.390+0000 7fa65f7fe700 1 -- 172.21.15.152:0/2738010887 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa654005040 con 0x7fa6600f6510 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.390+0000 7fa65f7fe700 1 --2- 172.21.15.152:0/2738010887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6600f6510 0x7fa660106730 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fa654013510 tx=0x7fa65400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.390+0000 7fa647fff700 1 -- 172.21.15.152:0/2738010887 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa654009320 con 0x7fa6600f6510 2023-12-06T03:58:16.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.390+0000 7fa647fff700 1 -- 172.21.15.152:0/2738010887 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa654013f00 con 0x7fa6600f6510 2023-12-06T03:58:16.394 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.390+0000 7fa665dce700 1 -- 172.21.15.152:0/2738010887 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa660106e20 con 0x7fa6600f6510 2023-12-06T03:58:16.394 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.390+0000 7fa647fff700 1 -- 172.21.15.152:0/2738010887 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa654014d90 con 0x7fa6600f6510 2023-12-06T03:58:16.394 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.391+0000 7fa665dce700 1 -- 172.21.15.152:0/2738010887 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6601072b0 con 0x7fa6600f6510 2023-12-06T03:58:16.394 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.391+0000 7fa647fff700 1 -- 172.21.15.152:0/2738010887 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa6540263a0 con 0x7fa6600f6510 2023-12-06T03:58:16.394 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.391+0000 7fa647fff700 1 --2- 172.21.15.152:0/2738010887 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa64802ff60 0x7fa648032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:16.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.391+0000 7fa665dce700 1 -- 172.21.15.152:0/2738010887 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa660045ad0 con 0x7fa6600f6510 2023-12-06T03:58:16.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.391+0000 7fa647fff700 1 -- 172.21.15.152:0/2738010887 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa654041380 con 0x7fa6600f6510 2023-12-06T03:58:16.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.391+0000 7fa65effd700 1 --2- 172.21.15.152:0/2738010887 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa64802ff60 0x7fa648032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:16.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.392+0000 7fa65effd700 1 --2- 172.21.15.152:0/2738010887 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa64802ff60 0x7fa648032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fa6500096a0 tx=0x7fa650006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:16.398 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.396+0000 7fa647fff700 1 -- 172.21.15.152:0/2738010887 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa654012330 con 0x7fa6600f6510 2023-12-06T03:58:16.605 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.603+0000 7fa665dce700 1 -- 172.21.15.152:0/2738010887 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa6600fc050 con 0x7fa6600f6510 2023-12-06T03:58:16.606 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.604+0000 7fa647fff700 1 -- 172.21.15.152:0/2738010887 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa654040090 con 0x7fa6600f6510 2023-12-06T03:58:16.606 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:16.606 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:16.607 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.605+0000 7fa665dce700 1 -- 172.21.15.152:0/2738010887 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa64802ff60 msgr2=0x7fa648032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:16.607 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.605+0000 7fa665dce700 1 --2- 172.21.15.152:0/2738010887 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa64802ff60 0x7fa648032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fa6500096a0 tx=0x7fa650006b40).stop 2023-12-06T03:58:16.607 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.605+0000 7fa665dce700 1 -- 172.21.15.152:0/2738010887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6600f6510 msgr2=0x7fa660106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:16.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.605+0000 7fa665dce700 1 --2- 172.21.15.152:0/2738010887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6600f6510 0x7fa660106730 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fa654013510 tx=0x7fa65400b040).stop 2023-12-06T03:58:16.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.605+0000 7fa665dce700 1 -- 172.21.15.152:0/2738010887 shutdown_connections 2023-12-06T03:58:16.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.605+0000 7fa665dce700 1 --2- 172.21.15.152:0/2738010887 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa64802ff60 0x7fa648032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:16.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.605+0000 7fa665dce700 1 --2- 172.21.15.152:0/2738010887 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa6600f6510 0x7fa660106730 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:16.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.605+0000 7fa665dce700 1 -- 172.21.15.152:0/2738010887 >> 172.21.15.152:0/2738010887 conn(0x7fa6600f1c70 msgr2=0x7fa6600f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:16.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.606+0000 7fa665dce700 1 -- 172.21.15.152:0/2738010887 shutdown_connections 2023-12-06T03:58:16.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:16.606+0000 7fa665dce700 1 -- 172.21.15.152:0/2738010887 wait complete. 2023-12-06T03:58:16.608 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:17.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:17 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:16.434714+0000 mgr.smithi132.gokxja (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:17.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:17 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:16.605393+0000 mon.smithi132 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.152:0/2738010887' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:18.001 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:18.001 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:19.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.099+0000 7f8aceea1700 1 -- 172.21.15.152:0/3543395650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ac80f6a00 msgr2=0x7f8ac80f6e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:19.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.099+0000 7f8aceea1700 1 --2- 172.21.15.152:0/3543395650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ac80f6a00 0x7f8ac80f6e20 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f8abc009fb0 tx=0x7f8abc009b30).stop 2023-12-06T03:58:19.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.099+0000 7f8aceea1700 1 -- 172.21.15.152:0/3543395650 shutdown_connections 2023-12-06T03:58:19.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.099+0000 7f8aceea1700 1 --2- 172.21.15.152:0/3543395650 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ac80f6a00 0x7f8ac80f6e20 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:19.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.099+0000 7f8aceea1700 1 -- 172.21.15.152:0/3543395650 >> 172.21.15.152:0/3543395650 conn(0x7f8ac80f0260 msgr2=0x7f8ac80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:19.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.099+0000 7f8aceea1700 1 -- 172.21.15.152:0/3543395650 shutdown_connections 2023-12-06T03:58:19.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.099+0000 7f8aceea1700 1 -- 172.21.15.152:0/3543395650 wait complete. 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.099+0000 7f8aceea1700 1 Processor -- start 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.099+0000 7f8aceea1700 1 -- start start 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.100+0000 7f8aceea1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ac80f6a00 0x7f8ac81093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.100+0000 7f8aceea1700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ac80f5290 con 0x7f8ac80f6a00 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.100+0000 7f8accc3d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ac80f6a00 0x7f8ac81093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.100+0000 7f8accc3d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ac80f6a00 0x7f8ac81093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:35758/0 (socket says 172.21.15.152:35758) 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.100+0000 7f8accc3d700 1 -- 172.21.15.152:0/3207249512 learned_addr learned my addr 172.21.15.152:0/3207249512 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.100+0000 7f8accc3d700 1 -- 172.21.15.152:0/3207249512 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8abc005040 con 0x7f8ac80f6a00 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.100+0000 7f8accc3d700 1 --2- 172.21.15.152:0/3207249512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ac80f6a00 0x7f8ac81093d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f8abc013510 tx=0x7f8abc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.100+0000 7f8ac57fa700 1 -- 172.21.15.152:0/3207249512 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8abc009320 con 0x7f8ac80f6a00 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.100+0000 7f8ac57fa700 1 -- 172.21.15.152:0/3207249512 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8abc013f00 con 0x7f8ac80f6a00 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.100+0000 7f8aceea1700 1 -- 172.21.15.152:0/3207249512 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ac80f5490 con 0x7f8ac80f6a00 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.100+0000 7f8ac57fa700 1 -- 172.21.15.152:0/3207249512 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8abc014dc0 con 0x7f8ac80f6a00 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.100+0000 7f8aceea1700 1 -- 172.21.15.152:0/3207249512 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ac8109a80 con 0x7f8ac80f6a00 2023-12-06T03:58:19.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.101+0000 7f8ac57fa700 1 -- 172.21.15.152:0/3207249512 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8abc014420 con 0x7f8ac80f6a00 2023-12-06T03:58:19.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.101+0000 7f8aceea1700 1 -- 172.21.15.152:0/3207249512 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ac8059100 con 0x7f8ac80f6a00 2023-12-06T03:58:19.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.101+0000 7f8ac57fa700 1 --2- 172.21.15.152:0/3207249512 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8ab802ff60 0x7f8ab8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:19.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.101+0000 7f8ac57fa700 1 -- 172.21.15.152:0/3207249512 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8abc0410f0 con 0x7f8ac80f6a00 2023-12-06T03:58:19.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.101+0000 7f8ac7fff700 1 --2- 172.21.15.152:0/3207249512 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8ab802ff60 0x7f8ab8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:19.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.102+0000 7f8ac7fff700 1 --2- 172.21.15.152:0/3207249512 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8ab802ff60 0x7f8ab8032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8ab40096a0 tx=0x7f8ab4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:19.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.106+0000 7f8ac57fa700 1 -- 172.21.15.152:0/3207249512 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8abc01ccf0 con 0x7f8ac80f6a00 2023-12-06T03:58:19.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.313+0000 7f8aceea1700 1 -- 172.21.15.152:0/3207249512 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ac8045ad0 con 0x7f8ac80f6a00 2023-12-06T03:58:19.316 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.314+0000 7f8ac57fa700 1 -- 172.21.15.152:0/3207249512 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8abc012330 con 0x7f8ac80f6a00 2023-12-06T03:58:19.316 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:19.316 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:19.317 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.316+0000 7f8aceea1700 1 -- 172.21.15.152:0/3207249512 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8ab802ff60 msgr2=0x7f8ab8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:19.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.316+0000 7f8aceea1700 1 --2- 172.21.15.152:0/3207249512 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8ab802ff60 0x7f8ab8032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f8ab40096a0 tx=0x7f8ab4006b40).stop 2023-12-06T03:58:19.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.316+0000 7f8aceea1700 1 -- 172.21.15.152:0/3207249512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ac80f6a00 msgr2=0x7f8ac81093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:19.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.316+0000 7f8aceea1700 1 --2- 172.21.15.152:0/3207249512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ac80f6a00 0x7f8ac81093d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f8abc013510 tx=0x7f8abc00b040).stop 2023-12-06T03:58:19.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.316+0000 7f8aceea1700 1 -- 172.21.15.152:0/3207249512 shutdown_connections 2023-12-06T03:58:19.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.316+0000 7f8aceea1700 1 --2- 172.21.15.152:0/3207249512 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8ab802ff60 0x7f8ab8032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:19.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.316+0000 7f8aceea1700 1 --2- 172.21.15.152:0/3207249512 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ac80f6a00 0x7f8ac81093d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:19.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.316+0000 7f8aceea1700 1 -- 172.21.15.152:0/3207249512 >> 172.21.15.152:0/3207249512 conn(0x7f8ac80f0260 msgr2=0x7f8ac80f0d60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:19.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.316+0000 7f8aceea1700 1 -- 172.21.15.152:0/3207249512 shutdown_connections 2023-12-06T03:58:19.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:19.316+0000 7f8aceea1700 1 -- 172.21.15.152:0/3207249512 wait complete. 2023-12-06T03:58:19.318 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:19.793 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:19 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:18.434956+0000 mgr.smithi132.gokxja (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:19.793 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:19 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:19.315051+0000 mon.smithi132 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.152:0/3207249512' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:20.703 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:20.704 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:21.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:19.681823+0000 mon.smithi132 (mon.0) 244 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' 2023-12-06T03:58:21.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T03:58:19.683086+0000 mgr.smithi132.gokxja (mgr.14162) 168 : cephadm [INF] Deploying daemon node-exporter.smithi132 on smithi132 2023-12-06T03:58:21.824 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.822+0000 7fc0267db700 1 -- 172.21.15.152:0/431723998 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0200f5410 msgr2=0x7fc0200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:21.824 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.822+0000 7fc0267db700 1 --2- 172.21.15.152:0/431723998 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0200f5410 0x7fc0200f5830 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fc014009fb0 tx=0x7fc014009b30).stop 2023-12-06T03:58:21.824 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.822+0000 7fc0267db700 1 -- 172.21.15.152:0/431723998 shutdown_connections 2023-12-06T03:58:21.824 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.822+0000 7fc0267db700 1 --2- 172.21.15.152:0/431723998 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0200f5410 0x7fc0200f5830 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:21.824 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.822+0000 7fc0267db700 1 -- 172.21.15.152:0/431723998 >> 172.21.15.152:0/431723998 conn(0x7fc0200f0b70 msgr2=0x7fc0200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:21.824 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.822+0000 7fc0267db700 1 -- 172.21.15.152:0/431723998 shutdown_connections 2023-12-06T03:58:21.824 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.822+0000 7fc0267db700 1 -- 172.21.15.152:0/431723998 wait complete. 2023-12-06T03:58:21.826 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.822+0000 7fc0267db700 1 Processor -- start 2023-12-06T03:58:21.826 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.823+0000 7fc0267db700 1 -- start start 2023-12-06T03:58:21.826 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.823+0000 7fc0267db700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0200f5410 0x7fc0200fdcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:21.826 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.823+0000 7fc0267db700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0200fe1b0 con 0x7fc0200f5410 2023-12-06T03:58:21.826 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.823+0000 7fc01ffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0200f5410 0x7fc0200fdcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:21.826 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.823+0000 7fc01ffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0200f5410 0x7fc0200fdcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:36560/0 (socket says 172.21.15.152:36560) 2023-12-06T03:58:21.826 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.823+0000 7fc01ffff700 1 -- 172.21.15.152:0/3881941729 learned_addr learned my addr 172.21.15.152:0/3881941729 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:21.826 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.823+0000 7fc01ffff700 1 -- 172.21.15.152:0/3881941729 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc014005040 con 0x7fc0200f5410 2023-12-06T03:58:21.826 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.823+0000 7fc01ffff700 1 --2- 172.21.15.152:0/3881941729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0200f5410 0x7fc0200fdcc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fc014013510 tx=0x7fc01400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:21.826 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.824+0000 7fc01cff9700 1 -- 172.21.15.152:0/3881941729 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc014009320 con 0x7fc0200f5410 2023-12-06T03:58:21.826 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.824+0000 7fc0267db700 1 -- 172.21.15.152:0/3881941729 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0200fe3b0 con 0x7fc0200f5410 2023-12-06T03:58:21.827 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.824+0000 7fc01cff9700 1 -- 172.21.15.152:0/3881941729 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc014013f00 con 0x7fc0200f5410 2023-12-06T03:58:21.827 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.824+0000 7fc01cff9700 1 -- 172.21.15.152:0/3881941729 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc014014d90 con 0x7fc0200f5410 2023-12-06T03:58:21.827 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.824+0000 7fc0267db700 1 -- 172.21.15.152:0/3881941729 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0200fe840 con 0x7fc0200f5410 2023-12-06T03:58:21.827 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.824+0000 7fc01cff9700 1 -- 172.21.15.152:0/3881941729 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0140263a0 con 0x7fc0200f5410 2023-12-06T03:58:21.827 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.825+0000 7fc01cff9700 1 --2- 172.21.15.152:0/3881941729 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc00802ff60 0x7fc008032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:21.828 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.825+0000 7fc01cff9700 1 -- 172.21.15.152:0/3881941729 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0140410d0 con 0x7fc0200f5410 2023-12-06T03:58:21.828 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.825+0000 7fc0267db700 1 -- 172.21.15.152:0/3881941729 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc020045ad0 con 0x7fc0200f5410 2023-12-06T03:58:21.828 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.825+0000 7fc01f7fe700 1 --2- 172.21.15.152:0/3881941729 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc00802ff60 0x7fc008032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:21.828 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.825+0000 7fc01f7fe700 1 --2- 172.21.15.152:0/3881941729 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc00802ff60 0x7fc008032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fc0100096a0 tx=0x7fc010006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:21.831 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:21.829+0000 7fc01cff9700 1 -- 172.21.15.152:0/3881941729 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc014012920 con 0x7fc0200f5410 2023-12-06T03:58:22.039 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:22.037+0000 7fc0267db700 1 -- 172.21.15.152:0/3881941729 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0200fae60 con 0x7fc0200f5410 2023-12-06T03:58:22.039 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:22.038+0000 7fc01cff9700 1 -- 172.21.15.152:0/3881941729 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc01402b020 con 0x7fc0200f5410 2023-12-06T03:58:22.040 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:22.040 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:22.042 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:22.039+0000 7fc0267db700 1 -- 172.21.15.152:0/3881941729 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc00802ff60 msgr2=0x7fc008032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:22.042 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:22.039+0000 7fc0267db700 1 --2- 172.21.15.152:0/3881941729 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc00802ff60 0x7fc008032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fc0100096a0 tx=0x7fc010006b40).stop 2023-12-06T03:58:22.042 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:22.039+0000 7fc0267db700 1 -- 172.21.15.152:0/3881941729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0200f5410 msgr2=0x7fc0200fdcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:22.042 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:22.039+0000 7fc0267db700 1 --2- 172.21.15.152:0/3881941729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0200f5410 0x7fc0200fdcc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fc014013510 tx=0x7fc01400b040).stop 2023-12-06T03:58:22.042 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:22.040+0000 7fc0267db700 1 -- 172.21.15.152:0/3881941729 shutdown_connections 2023-12-06T03:58:22.043 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:22.040+0000 7fc0267db700 1 --2- 172.21.15.152:0/3881941729 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc00802ff60 0x7fc008032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:22.043 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:22.040+0000 7fc0267db700 1 --2- 172.21.15.152:0/3881941729 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc0200f5410 0x7fc0200fdcc0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:22.043 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:22.040+0000 7fc0267db700 1 -- 172.21.15.152:0/3881941729 >> 172.21.15.152:0/3881941729 conn(0x7fc0200f0b70 msgr2=0x7fc02005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:22.043 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:22.040+0000 7fc0267db700 1 -- 172.21.15.152:0/3881941729 shutdown_connections 2023-12-06T03:58:22.043 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:22.040+0000 7fc0267db700 1 -- 172.21.15.152:0/3881941729 wait complete. 2023-12-06T03:58:22.043 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:22.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:20.435237+0000 mgr.smithi132.gokxja (mgr.14162) 169 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:23.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:22 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:22.039481+0000 mon.smithi132 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.152:0/3881941729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:23.438 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:23.439 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:24.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:22.435567+0000 mgr.smithi132.gokxja (mgr.14162) 170 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:24.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.597+0000 7f98745eb700 1 -- 172.21.15.152:0/3201903840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f986c0f6510 msgr2=0x7f986c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:24.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.597+0000 7f98745eb700 1 --2- 172.21.15.152:0/3201903840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f986c0f6510 0x7f986c0f6930 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f985c009fb0 tx=0x7f985c009b30).stop 2023-12-06T03:58:24.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.597+0000 7f98745eb700 1 -- 172.21.15.152:0/3201903840 shutdown_connections 2023-12-06T03:58:24.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.597+0000 7f98745eb700 1 --2- 172.21.15.152:0/3201903840 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f986c0f6510 0x7f986c0f6930 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:24.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.597+0000 7f98745eb700 1 -- 172.21.15.152:0/3201903840 >> 172.21.15.152:0/3201903840 conn(0x7f986c0f1c70 msgr2=0x7f986c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:24.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.597+0000 7f98745eb700 1 -- 172.21.15.152:0/3201903840 shutdown_connections 2023-12-06T03:58:24.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.597+0000 7f98745eb700 1 -- 172.21.15.152:0/3201903840 wait complete. 2023-12-06T03:58:24.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.598+0000 7f98745eb700 1 Processor -- start 2023-12-06T03:58:24.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.598+0000 7f98745eb700 1 -- start start 2023-12-06T03:58:24.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.598+0000 7f98745eb700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f986c0f6510 0x7f986c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:24.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.598+0000 7f98745eb700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f986c069e20 con 0x7f986c0f6510 2023-12-06T03:58:24.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.598+0000 7f9872387700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f986c0f6510 0x7f986c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:24.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.598+0000 7f9872387700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f986c0f6510 0x7f986c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:36564/0 (socket says 172.21.15.152:36564) 2023-12-06T03:58:24.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.598+0000 7f9872387700 1 -- 172.21.15.152:0/2848925437 learned_addr learned my addr 172.21.15.152:0/2848925437 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:24.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.599+0000 7f9872387700 1 -- 172.21.15.152:0/2848925437 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f985c005040 con 0x7f986c0f6510 2023-12-06T03:58:24.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.599+0000 7f9872387700 1 --2- 172.21.15.152:0/2848925437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f986c0f6510 0x7f986c06b9a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f985c013510 tx=0x7f985c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:24.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.599+0000 7f9862ffd700 1 -- 172.21.15.152:0/2848925437 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f985c009320 con 0x7f986c0f6510 2023-12-06T03:58:24.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.599+0000 7f9862ffd700 1 -- 172.21.15.152:0/2848925437 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f985c013f00 con 0x7f986c0f6510 2023-12-06T03:58:24.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.599+0000 7f98745eb700 1 -- 172.21.15.152:0/2848925437 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f986c069fc0 con 0x7f986c0f6510 2023-12-06T03:58:24.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.599+0000 7f9862ffd700 1 -- 172.21.15.152:0/2848925437 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f985c014d90 con 0x7f986c0f6510 2023-12-06T03:58:24.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.599+0000 7f98745eb700 1 -- 172.21.15.152:0/2848925437 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f986c06a450 con 0x7f986c0f6510 2023-12-06T03:58:24.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.600+0000 7f9862ffd700 1 -- 172.21.15.152:0/2848925437 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f985c014420 con 0x7f986c0f6510 2023-12-06T03:58:24.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.600+0000 7f98745eb700 1 -- 172.21.15.152:0/2848925437 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f986c059100 con 0x7f986c0f6510 2023-12-06T03:58:24.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.600+0000 7f9862ffd700 1 --2- 172.21.15.152:0/2848925437 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f985802ffb0 0x7f9858032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:24.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.600+0000 7f9862ffd700 1 -- 172.21.15.152:0/2848925437 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f985c0410b0 con 0x7f986c0f6510 2023-12-06T03:58:24.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.600+0000 7f9871b86700 1 --2- 172.21.15.152:0/2848925437 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f985802ffb0 0x7f9858032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:24.604 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.600+0000 7f9871b86700 1 --2- 172.21.15.152:0/2848925437 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f985802ffb0 0x7f9858032470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f98680096a0 tx=0x7f9868006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:24.606 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.604+0000 7f9862ffd700 1 -- 172.21.15.152:0/2848925437 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f985c06d2b0 con 0x7f986c0f6510 2023-12-06T03:58:24.813 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.811+0000 7f98745eb700 1 -- 172.21.15.152:0/2848925437 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f986c0fbf60 con 0x7f986c0f6510 2023-12-06T03:58:24.814 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.811+0000 7f9862ffd700 1 -- 172.21.15.152:0/2848925437 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f985c0095f0 con 0x7f986c0f6510 2023-12-06T03:58:24.815 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:24.815 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:24.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.813+0000 7f98745eb700 1 -- 172.21.15.152:0/2848925437 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f985802ffb0 msgr2=0x7f9858032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:24.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.813+0000 7f98745eb700 1 --2- 172.21.15.152:0/2848925437 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f985802ffb0 0x7f9858032470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f98680096a0 tx=0x7f9868006b40).stop 2023-12-06T03:58:24.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.813+0000 7f98745eb700 1 -- 172.21.15.152:0/2848925437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f986c0f6510 msgr2=0x7f986c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:24.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.813+0000 7f98745eb700 1 --2- 172.21.15.152:0/2848925437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f986c0f6510 0x7f986c06b9a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f985c013510 tx=0x7f985c00b040).stop 2023-12-06T03:58:24.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.813+0000 7f98745eb700 1 -- 172.21.15.152:0/2848925437 shutdown_connections 2023-12-06T03:58:24.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.813+0000 7f98745eb700 1 --2- 172.21.15.152:0/2848925437 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f985802ffb0 0x7f9858032470 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:24.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.813+0000 7f98745eb700 1 --2- 172.21.15.152:0/2848925437 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f986c0f6510 0x7f986c06b9a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:24.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.813+0000 7f98745eb700 1 -- 172.21.15.152:0/2848925437 >> 172.21.15.152:0/2848925437 conn(0x7f986c0f1c70 msgr2=0x7f986c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:24.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.813+0000 7f98745eb700 1 -- 172.21.15.152:0/2848925437 shutdown_connections 2023-12-06T03:58:24.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:24.813+0000 7f98745eb700 1 -- 172.21.15.152:0/2848925437 wait complete. 2023-12-06T03:58:24.816 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:26.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:24.435923+0000 mgr.smithi132.gokxja (mgr.14162) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:26.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:24.812827+0000 mon.smithi132 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.152:0/2848925437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:26.196 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:26.196 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:27.334 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.332+0000 7fec1afda700 1 -- 172.21.15.152:0/2149832026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec140f5410 msgr2=0x7fec140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:27.334 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.332+0000 7fec1afda700 1 --2- 172.21.15.152:0/2149832026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec140f5410 0x7fec140f5830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7fec08009fb0 tx=0x7fec08009b30).stop 2023-12-06T03:58:27.334 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.332+0000 7fec1afda700 1 -- 172.21.15.152:0/2149832026 shutdown_connections 2023-12-06T03:58:27.334 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.332+0000 7fec1afda700 1 --2- 172.21.15.152:0/2149832026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec140f5410 0x7fec140f5830 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:27.335 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.332+0000 7fec1afda700 1 -- 172.21.15.152:0/2149832026 >> 172.21.15.152:0/2149832026 conn(0x7fec140f0b70 msgr2=0x7fec140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:27.335 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.332+0000 7fec1afda700 1 -- 172.21.15.152:0/2149832026 shutdown_connections 2023-12-06T03:58:27.335 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.332+0000 7fec1afda700 1 -- 172.21.15.152:0/2149832026 wait complete. 2023-12-06T03:58:27.335 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.332+0000 7fec1afda700 1 Processor -- start 2023-12-06T03:58:27.335 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.332+0000 7fec1afda700 1 -- start start 2023-12-06T03:58:27.335 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.333+0000 7fec1afda700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec140f5410 0x7fec14106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:27.335 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.333+0000 7fec1afda700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec14106c10 con 0x7fec140f5410 2023-12-06T03:58:27.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.333+0000 7fec18d76700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec140f5410 0x7fec14106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:27.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.333+0000 7fec18d76700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec140f5410 0x7fec14106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:36572/0 (socket says 172.21.15.152:36572) 2023-12-06T03:58:27.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.333+0000 7fec18d76700 1 -- 172.21.15.152:0/2346032559 learned_addr learned my addr 172.21.15.152:0/2346032559 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:27.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.333+0000 7fec18d76700 1 -- 172.21.15.152:0/2346032559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec08005040 con 0x7fec140f5410 2023-12-06T03:58:27.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.333+0000 7fec18d76700 1 --2- 172.21.15.152:0/2346032559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec140f5410 0x7fec14106720 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fec08013510 tx=0x7fec0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:27.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.334+0000 7fec117fa700 1 -- 172.21.15.152:0/2346032559 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec08009320 con 0x7fec140f5410 2023-12-06T03:58:27.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.334+0000 7fec117fa700 1 -- 172.21.15.152:0/2346032559 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fec08013f00 con 0x7fec140f5410 2023-12-06T03:58:27.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.334+0000 7fec1afda700 1 -- 172.21.15.152:0/2346032559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec14106e10 con 0x7fec140f5410 2023-12-06T03:58:27.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.334+0000 7fec117fa700 1 -- 172.21.15.152:0/2346032559 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec08014d90 con 0x7fec140f5410 2023-12-06T03:58:27.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.334+0000 7fec1afda700 1 -- 172.21.15.152:0/2346032559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec141072a0 con 0x7fec140f5410 2023-12-06T03:58:27.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.335+0000 7fec117fa700 1 -- 172.21.15.152:0/2346032559 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fec08014420 con 0x7fec140f5410 2023-12-06T03:58:27.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.335+0000 7fec1afda700 1 -- 172.21.15.152:0/2346032559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec14045ad0 con 0x7fec140f5410 2023-12-06T03:58:27.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.335+0000 7fec117fa700 1 --2- 172.21.15.152:0/2346032559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fec0402ff60 0x7fec04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:27.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.335+0000 7fec117fa700 1 -- 172.21.15.152:0/2346032559 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fec08041000 con 0x7fec140f5410 2023-12-06T03:58:27.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.335+0000 7fec13fff700 1 --2- 172.21.15.152:0/2346032559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fec0402ff60 0x7fec04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:27.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.335+0000 7fec13fff700 1 --2- 172.21.15.152:0/2346032559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fec0402ff60 0x7fec04032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fec000096a0 tx=0x7fec00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:27.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.340+0000 7fec117fa700 1 -- 172.21.15.152:0/2346032559 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fec080095f0 con 0x7fec140f5410 2023-12-06T03:58:27.548 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.546+0000 7fec1afda700 1 -- 172.21.15.152:0/2346032559 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec14108150 con 0x7fec140f5410 2023-12-06T03:58:27.548 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.547+0000 7fec117fa700 1 -- 172.21.15.152:0/2346032559 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fec0801db70 con 0x7fec140f5410 2023-12-06T03:58:27.549 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:27.549 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:27.550 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.548+0000 7fec1afda700 1 -- 172.21.15.152:0/2346032559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fec0402ff60 msgr2=0x7fec04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:27.550 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.548+0000 7fec1afda700 1 --2- 172.21.15.152:0/2346032559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fec0402ff60 0x7fec04032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fec000096a0 tx=0x7fec00006b40).stop 2023-12-06T03:58:27.550 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.548+0000 7fec1afda700 1 -- 172.21.15.152:0/2346032559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec140f5410 msgr2=0x7fec14106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:27.550 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.548+0000 7fec1afda700 1 --2- 172.21.15.152:0/2346032559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec140f5410 0x7fec14106720 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fec08013510 tx=0x7fec0800b040).stop 2023-12-06T03:58:27.550 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.548+0000 7fec1afda700 1 -- 172.21.15.152:0/2346032559 shutdown_connections 2023-12-06T03:58:27.550 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.548+0000 7fec1afda700 1 --2- 172.21.15.152:0/2346032559 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fec0402ff60 0x7fec04032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:27.551 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.548+0000 7fec1afda700 1 --2- 172.21.15.152:0/2346032559 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fec140f5410 0x7fec14106720 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:27.551 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.548+0000 7fec1afda700 1 -- 172.21.15.152:0/2346032559 >> 172.21.15.152:0/2346032559 conn(0x7fec140f0b70 msgr2=0x7fec1405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:27.551 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.549+0000 7fec1afda700 1 -- 172.21.15.152:0/2346032559 shutdown_connections 2023-12-06T03:58:27.551 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:27.549+0000 7fec1afda700 1 -- 172.21.15.152:0/2346032559 wait complete. 2023-12-06T03:58:27.551 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:28.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:26.436284+0000 mgr.smithi132.gokxja (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:28.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:27.548116+0000 mon.smithi132 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.152:0/2346032559' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:28.880 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:28.880 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:29.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.990+0000 7f67e94fe700 1 -- 172.21.15.152:0/1118935781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67e40f5410 msgr2=0x7f67e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:29.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.990+0000 7f67e94fe700 1 --2- 172.21.15.152:0/1118935781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67e40f5410 0x7f67e40f5830 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f67d8009fb0 tx=0x7f67d8009b30).stop 2023-12-06T03:58:29.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.990+0000 7f67e94fe700 1 -- 172.21.15.152:0/1118935781 shutdown_connections 2023-12-06T03:58:29.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.990+0000 7f67e94fe700 1 --2- 172.21.15.152:0/1118935781 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67e40f5410 0x7f67e40f5830 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:29.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.990+0000 7f67e94fe700 1 -- 172.21.15.152:0/1118935781 >> 172.21.15.152:0/1118935781 conn(0x7f67e40f0b70 msgr2=0x7f67e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:29.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.990+0000 7f67e94fe700 1 -- 172.21.15.152:0/1118935781 shutdown_connections 2023-12-06T03:58:29.997 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.990+0000 7f67e94fe700 1 -- 172.21.15.152:0/1118935781 wait complete. 2023-12-06T03:58:29.997 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.991+0000 7f67e94fe700 1 Processor -- start 2023-12-06T03:58:29.997 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.991+0000 7f67e94fe700 1 -- start start 2023-12-06T03:58:29.997 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.991+0000 7f67e94fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67e40f5410 0x7f67e41093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:29.997 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.991+0000 7f67e94fe700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67e4060280 con 0x7f67e40f5410 2023-12-06T03:58:29.997 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.991+0000 7f67e2ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67e40f5410 0x7f67e41093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:29.997 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.991+0000 7f67e2ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67e40f5410 0x7f67e41093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:53786/0 (socket says 172.21.15.152:53786) 2023-12-06T03:58:29.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.991+0000 7f67e2ffd700 1 -- 172.21.15.152:0/3259038886 learned_addr learned my addr 172.21.15.152:0/3259038886 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:29.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.991+0000 7f67e2ffd700 1 -- 172.21.15.152:0/3259038886 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67d8005040 con 0x7f67e40f5410 2023-12-06T03:58:29.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.991+0000 7f67e2ffd700 1 --2- 172.21.15.152:0/3259038886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67e40f5410 0x7f67e41093f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f67d8013510 tx=0x7f67d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:29.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.992+0000 7f67cb7fe700 1 -- 172.21.15.152:0/3259038886 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67d8009320 con 0x7f67e40f5410 2023-12-06T03:58:29.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.992+0000 7f67e94fe700 1 -- 172.21.15.152:0/3259038886 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67e4060480 con 0x7f67e40f5410 2023-12-06T03:58:29.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.992+0000 7f67cb7fe700 1 -- 172.21.15.152:0/3259038886 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67d8013f00 con 0x7f67e40f5410 2023-12-06T03:58:29.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.992+0000 7f67cb7fe700 1 -- 172.21.15.152:0/3259038886 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67d8014d90 con 0x7f67e40f5410 2023-12-06T03:58:29.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.992+0000 7f67e94fe700 1 -- 172.21.15.152:0/3259038886 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67e4109b20 con 0x7f67e40f5410 2023-12-06T03:58:29.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.992+0000 7f67cb7fe700 1 -- 172.21.15.152:0/3259038886 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67d8014420 con 0x7f67e40f5410 2023-12-06T03:58:29.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.993+0000 7f67e94fe700 1 -- 172.21.15.152:0/3259038886 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67e4045ad0 con 0x7f67e40f5410 2023-12-06T03:58:29.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.993+0000 7f67cb7fe700 1 --2- 172.21.15.152:0/3259038886 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f67cc02ff60 0x7f67cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:29.999 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.993+0000 7f67cb7fe700 1 -- 172.21.15.152:0/3259038886 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67d80410c0 con 0x7f67e40f5410 2023-12-06T03:58:29.999 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.993+0000 7f67e27fc700 1 --2- 172.21.15.152:0/3259038886 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f67cc02ff60 0x7f67cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:29.999 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.993+0000 7f67e27fc700 1 --2- 172.21.15.152:0/3259038886 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f67cc02ff60 0x7f67cc032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f67d40096a0 tx=0x7f67d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:30.000 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:29.997+0000 7f67cb7fe700 1 -- 172.21.15.152:0/3259038886 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67d80095f0 con 0x7f67e40f5410 2023-12-06T03:58:30.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:28.436650+0000 mgr.smithi132.gokxja (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:30.208 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:30.206+0000 7f67e94fe700 1 -- 172.21.15.152:0/3259038886 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67e410a340 con 0x7f67e40f5410 2023-12-06T03:58:30.209 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:30.207+0000 7f67cb7fe700 1 -- 172.21.15.152:0/3259038886 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f67d801db70 con 0x7f67e40f5410 2023-12-06T03:58:30.209 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:30.209 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:30.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:30.209+0000 7f67e94fe700 1 -- 172.21.15.152:0/3259038886 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f67cc02ff60 msgr2=0x7f67cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:30.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:30.209+0000 7f67e94fe700 1 --2- 172.21.15.152:0/3259038886 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f67cc02ff60 0x7f67cc032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f67d40096a0 tx=0x7f67d4006b40).stop 2023-12-06T03:58:30.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:30.209+0000 7f67e94fe700 1 -- 172.21.15.152:0/3259038886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67e40f5410 msgr2=0x7f67e41093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:30.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:30.209+0000 7f67e94fe700 1 --2- 172.21.15.152:0/3259038886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67e40f5410 0x7f67e41093f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f67d8013510 tx=0x7f67d800b040).stop 2023-12-06T03:58:30.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:30.209+0000 7f67e94fe700 1 -- 172.21.15.152:0/3259038886 shutdown_connections 2023-12-06T03:58:30.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:30.209+0000 7f67e94fe700 1 --2- 172.21.15.152:0/3259038886 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f67cc02ff60 0x7f67cc032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:30.212 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:30.209+0000 7f67e94fe700 1 --2- 172.21.15.152:0/3259038886 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f67e40f5410 0x7f67e41093f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:30.212 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:30.209+0000 7f67e94fe700 1 -- 172.21.15.152:0/3259038886 >> 172.21.15.152:0/3259038886 conn(0x7f67e40f0b70 msgr2=0x7f67e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:30.212 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:30.209+0000 7f67e94fe700 1 -- 172.21.15.152:0/3259038886 shutdown_connections 2023-12-06T03:58:30.212 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:30.209+0000 7f67e94fe700 1 -- 172.21.15.152:0/3259038886 wait complete. 2023-12-06T03:58:30.212 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:31.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:30 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:30.208489+0000 mon.smithi132 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.152:0/3259038886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:31.608 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:31.608 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:32.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:31 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:30.437019+0000 mgr.smithi132.gokxja (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:32.730 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.727+0000 7f841f866700 1 -- 172.21.15.152:0/1611294272 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84180f47c0 msgr2=0x7f84180f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:32.731 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.727+0000 7f841f866700 1 --2- 172.21.15.152:0/1611294272 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84180f47c0 0x7f84180f4be0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f8408009fb0 tx=0x7f8408009b30).stop 2023-12-06T03:58:32.731 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.727+0000 7f841f866700 1 -- 172.21.15.152:0/1611294272 shutdown_connections 2023-12-06T03:58:32.731 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.727+0000 7f841f866700 1 --2- 172.21.15.152:0/1611294272 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84180f47c0 0x7f84180f4be0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:32.731 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.727+0000 7f841f866700 1 -- 172.21.15.152:0/1611294272 >> 172.21.15.152:0/1611294272 conn(0x7f84180f02e0 msgr2=0x7f84180f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:32.731 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.727+0000 7f841f866700 1 -- 172.21.15.152:0/1611294272 shutdown_connections 2023-12-06T03:58:32.731 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.727+0000 7f841f866700 1 -- 172.21.15.152:0/1611294272 wait complete. 2023-12-06T03:58:32.731 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.728+0000 7f841f866700 1 Processor -- start 2023-12-06T03:58:32.731 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.728+0000 7f841f866700 1 -- start start 2023-12-06T03:58:32.731 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.728+0000 7f841f866700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84180f47c0 0x7f8418106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:32.731 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.728+0000 7f841f866700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8418106bf0 con 0x7f84180f47c0 2023-12-06T03:58:32.731 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.728+0000 7f841d602700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84180f47c0 0x7f8418106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:32.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.728+0000 7f841d602700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84180f47c0 0x7f8418106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:53812/0 (socket says 172.21.15.152:53812) 2023-12-06T03:58:32.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.728+0000 7f841d602700 1 -- 172.21.15.152:0/3205658469 learned_addr learned my addr 172.21.15.152:0/3205658469 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:32.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.729+0000 7f841d602700 1 -- 172.21.15.152:0/3205658469 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8408005040 con 0x7f84180f47c0 2023-12-06T03:58:32.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.729+0000 7f841d602700 1 --2- 172.21.15.152:0/3205658469 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84180f47c0 0x7f8418106700 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f8408013510 tx=0x7f840800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:32.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.729+0000 7f840dffb700 1 -- 172.21.15.152:0/3205658469 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8408009320 con 0x7f84180f47c0 2023-12-06T03:58:32.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.729+0000 7f841f866700 1 -- 172.21.15.152:0/3205658469 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8418106df0 con 0x7f84180f47c0 2023-12-06T03:58:32.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.729+0000 7f840dffb700 1 -- 172.21.15.152:0/3205658469 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8408013f00 con 0x7f84180f47c0 2023-12-06T03:58:32.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.729+0000 7f840dffb700 1 -- 172.21.15.152:0/3205658469 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8408014d90 con 0x7f84180f47c0 2023-12-06T03:58:32.732 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.729+0000 7f841f866700 1 -- 172.21.15.152:0/3205658469 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8418107280 con 0x7f84180f47c0 2023-12-06T03:58:32.733 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.730+0000 7f840dffb700 1 -- 172.21.15.152:0/3205658469 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84080095f0 con 0x7f84180f47c0 2023-12-06T03:58:32.733 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.730+0000 7f841f866700 1 -- 172.21.15.152:0/3205658469 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8418045ad0 con 0x7f84180f47c0 2023-12-06T03:58:32.733 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.730+0000 7f840dffb700 1 --2- 172.21.15.152:0/3205658469 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f840402ff60 0x7f8404032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:32.733 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.730+0000 7f840dffb700 1 -- 172.21.15.152:0/3205658469 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8408040fc0 con 0x7f84180f47c0 2023-12-06T03:58:32.733 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.730+0000 7f841ce01700 1 --2- 172.21.15.152:0/3205658469 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f840402ff60 0x7f8404032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:32.733 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.731+0000 7f841ce01700 1 --2- 172.21.15.152:0/3205658469 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f840402ff60 0x7f8404032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f84140096a0 tx=0x7f8414006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:32.737 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.735+0000 7f840dffb700 1 -- 172.21.15.152:0/3205658469 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8408014420 con 0x7f84180f47c0 2023-12-06T03:58:32.942 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.940+0000 7f841f866700 1 -- 172.21.15.152:0/3205658469 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84181080d0 con 0x7f84180f47c0 2023-12-06T03:58:32.943 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.941+0000 7f840dffb700 1 -- 172.21.15.152:0/3205658469 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8408010230 con 0x7f84180f47c0 2023-12-06T03:58:32.943 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:32.943 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:32.945 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.943+0000 7f841f866700 1 -- 172.21.15.152:0/3205658469 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f840402ff60 msgr2=0x7f8404032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:32.945 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.943+0000 7f841f866700 1 --2- 172.21.15.152:0/3205658469 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f840402ff60 0x7f8404032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f84140096a0 tx=0x7f8414006b40).stop 2023-12-06T03:58:32.945 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.943+0000 7f841f866700 1 -- 172.21.15.152:0/3205658469 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84180f47c0 msgr2=0x7f8418106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:32.945 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.943+0000 7f841f866700 1 --2- 172.21.15.152:0/3205658469 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84180f47c0 0x7f8418106700 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f8408013510 tx=0x7f840800b040).stop 2023-12-06T03:58:32.945 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.943+0000 7f841f866700 1 -- 172.21.15.152:0/3205658469 shutdown_connections 2023-12-06T03:58:32.945 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.943+0000 7f841f866700 1 --2- 172.21.15.152:0/3205658469 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f840402ff60 0x7f8404032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:32.945 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.943+0000 7f841f866700 1 --2- 172.21.15.152:0/3205658469 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f84180f47c0 0x7f8418106700 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:32.945 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.943+0000 7f841f866700 1 -- 172.21.15.152:0/3205658469 >> 172.21.15.152:0/3205658469 conn(0x7f84180f02e0 msgr2=0x7f84180f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:32.945 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.943+0000 7f841f866700 1 -- 172.21.15.152:0/3205658469 shutdown_connections 2023-12-06T03:58:32.945 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:32.943+0000 7f841f866700 1 -- 172.21.15.152:0/3205658469 wait complete. 2023-12-06T03:58:32.946 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:34.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:33 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:32.437290+0000 mgr.smithi132.gokxja (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:34.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:33 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:32.942648+0000 mon.smithi132 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.152:0/3205658469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:34.311 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:34.311 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:35.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.410+0000 7fdf21225700 1 -- 172.21.15.152:0/3325426151 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf1c0f6510 msgr2=0x7fdf1c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:35.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.411+0000 7fdf21225700 1 --2- 172.21.15.152:0/3325426151 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c0f6930 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fdf0c009fb0 tx=0x7fdf0c009b30).stop 2023-12-06T03:58:35.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.411+0000 7fdf21225700 1 -- 172.21.15.152:0/3325426151 shutdown_connections 2023-12-06T03:58:35.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.411+0000 7fdf21225700 1 --2- 172.21.15.152:0/3325426151 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c0f6930 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:35.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.411+0000 7fdf21225700 1 -- 172.21.15.152:0/3325426151 >> 172.21.15.152:0/3325426151 conn(0x7fdf1c0f1c70 msgr2=0x7fdf1c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:35.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.411+0000 7fdf21225700 1 -- 172.21.15.152:0/3325426151 shutdown_connections 2023-12-06T03:58:35.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.411+0000 7fdf21225700 1 -- 172.21.15.152:0/3325426151 wait complete. 2023-12-06T03:58:35.415 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.411+0000 7fdf21225700 1 Processor -- start 2023-12-06T03:58:35.415 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.411+0000 7fdf21225700 1 -- start start 2023-12-06T03:58:35.415 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.411+0000 7fdf21225700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:35.415 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.411+0000 7fdf21225700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c106c20 con 0x7fdf1c0f6510 2023-12-06T03:58:35.415 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.412+0000 7fdf1ad9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:35.415 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.412+0000 7fdf1ad9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:53828/0 (socket says 172.21.15.152:53828) 2023-12-06T03:58:35.415 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.412+0000 7fdf1ad9d700 1 -- 172.21.15.152:0/4188636087 learned_addr learned my addr 172.21.15.152:0/4188636087 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:35.415 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.412+0000 7fdf1ad9d700 1 -- 172.21.15.152:0/4188636087 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf0c005040 con 0x7fdf1c0f6510 2023-12-06T03:58:35.415 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.412+0000 7fdf1ad9d700 1 --2- 172.21.15.152:0/4188636087 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c106730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fdf0c013510 tx=0x7fdf0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:35.415 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.412+0000 7fdf037fe700 1 -- 172.21.15.152:0/4188636087 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf0c009320 con 0x7fdf1c0f6510 2023-12-06T03:58:35.416 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.412+0000 7fdf037fe700 1 -- 172.21.15.152:0/4188636087 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf0c013f00 con 0x7fdf1c0f6510 2023-12-06T03:58:35.416 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.412+0000 7fdf21225700 1 -- 172.21.15.152:0/4188636087 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1c106e20 con 0x7fdf1c0f6510 2023-12-06T03:58:35.416 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.412+0000 7fdf037fe700 1 -- 172.21.15.152:0/4188636087 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf0c014d90 con 0x7fdf1c0f6510 2023-12-06T03:58:35.416 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.412+0000 7fdf21225700 1 -- 172.21.15.152:0/4188636087 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1c1072b0 con 0x7fdf1c0f6510 2023-12-06T03:58:35.416 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.413+0000 7fdf037fe700 1 -- 172.21.15.152:0/4188636087 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf0c0263a0 con 0x7fdf1c0f6510 2023-12-06T03:58:35.416 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.413+0000 7fdf21225700 1 -- 172.21.15.152:0/4188636087 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1c045ad0 con 0x7fdf1c0f6510 2023-12-06T03:58:35.416 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.413+0000 7fdf037fe700 1 --2- 172.21.15.152:0/4188636087 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf0402ff60 0x7fdf04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:35.416 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.413+0000 7fdf037fe700 1 -- 172.21.15.152:0/4188636087 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf0c041390 con 0x7fdf1c0f6510 2023-12-06T03:58:35.417 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.414+0000 7fdf1a59c700 1 --2- 172.21.15.152:0/4188636087 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf0402ff60 0x7fdf04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:35.417 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.414+0000 7fdf1a59c700 1 --2- 172.21.15.152:0/4188636087 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf0402ff60 0x7fdf04032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fdf100096a0 tx=0x7fdf10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:35.420 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.418+0000 7fdf037fe700 1 -- 172.21.15.152:0/4188636087 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf0c01d380 con 0x7fdf1c0f6510 2023-12-06T03:58:35.626 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.624+0000 7fdf21225700 1 -- 172.21.15.152:0/4188636087 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf1c0fbf60 con 0x7fdf1c0f6510 2023-12-06T03:58:35.627 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.625+0000 7fdf037fe700 1 -- 172.21.15.152:0/4188636087 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdf0c014420 con 0x7fdf1c0f6510 2023-12-06T03:58:35.627 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:35.627 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:35.628 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.626+0000 7fdf21225700 1 -- 172.21.15.152:0/4188636087 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf0402ff60 msgr2=0x7fdf04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:35.629 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.626+0000 7fdf21225700 1 --2- 172.21.15.152:0/4188636087 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf0402ff60 0x7fdf04032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fdf100096a0 tx=0x7fdf10006b40).stop 2023-12-06T03:58:35.629 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.626+0000 7fdf21225700 1 -- 172.21.15.152:0/4188636087 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf1c0f6510 msgr2=0x7fdf1c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:35.629 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.626+0000 7fdf21225700 1 --2- 172.21.15.152:0/4188636087 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c106730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fdf0c013510 tx=0x7fdf0c00b040).stop 2023-12-06T03:58:35.629 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.627+0000 7fdf21225700 1 -- 172.21.15.152:0/4188636087 shutdown_connections 2023-12-06T03:58:35.629 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.627+0000 7fdf21225700 1 --2- 172.21.15.152:0/4188636087 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf0402ff60 0x7fdf04032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:35.629 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.627+0000 7fdf21225700 1 --2- 172.21.15.152:0/4188636087 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf1c0f6510 0x7fdf1c106730 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:35.629 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.627+0000 7fdf21225700 1 -- 172.21.15.152:0/4188636087 >> 172.21.15.152:0/4188636087 conn(0x7fdf1c0f1c70 msgr2=0x7fdf1c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:35.629 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.627+0000 7fdf21225700 1 -- 172.21.15.152:0/4188636087 shutdown_connections 2023-12-06T03:58:35.629 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:35.627+0000 7fdf21225700 1 -- 172.21.15.152:0/4188636087 wait complete. 2023-12-06T03:58:35.629 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:36.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:35 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:34.437617+0000 mgr.smithi132.gokxja (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:36.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:35 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:35.626197+0000 mon.smithi132 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.152:0/4188636087' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:37.000 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:37.000 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:38.079 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.077+0000 7fc4369d3700 1 -- 172.21.15.152:0/3708278694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f6510 msgr2=0x7fc4300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.077+0000 7fc4369d3700 1 --2- 172.21.15.152:0/3708278694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f6510 0x7fc4300f6930 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fc420009fb0 tx=0x7fc420009b30).stop 2023-12-06T03:58:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.077+0000 7fc4369d3700 1 -- 172.21.15.152:0/3708278694 shutdown_connections 2023-12-06T03:58:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.077+0000 7fc4369d3700 1 --2- 172.21.15.152:0/3708278694 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f6510 0x7fc4300f6930 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.077+0000 7fc4369d3700 1 -- 172.21.15.152:0/3708278694 >> 172.21.15.152:0/3708278694 conn(0x7fc4300f1c70 msgr2=0x7fc4300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.077+0000 7fc4369d3700 1 -- 172.21.15.152:0/3708278694 shutdown_connections 2023-12-06T03:58:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.077+0000 7fc4369d3700 1 -- 172.21.15.152:0/3708278694 wait complete. 2023-12-06T03:58:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.078+0000 7fc4369d3700 1 Processor -- start 2023-12-06T03:58:38.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.078+0000 7fc4369d3700 1 -- start start 2023-12-06T03:58:38.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.078+0000 7fc4369d3700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f6510 0x7fc430106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:38.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.078+0000 7fc4369d3700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc430106c20 con 0x7fc4300f6510 2023-12-06T03:58:38.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.079+0000 7fc42ffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f6510 0x7fc430106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:38.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.079+0000 7fc42ffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f6510 0x7fc430106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:53838/0 (socket says 172.21.15.152:53838) 2023-12-06T03:58:38.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.079+0000 7fc42ffff700 1 -- 172.21.15.152:0/2891427266 learned_addr learned my addr 172.21.15.152:0/2891427266 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:38.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.079+0000 7fc42ffff700 1 -- 172.21.15.152:0/2891427266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc420005040 con 0x7fc4300f6510 2023-12-06T03:58:38.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.079+0000 7fc42ffff700 1 --2- 172.21.15.152:0/2891427266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f6510 0x7fc430106730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fc420013510 tx=0x7fc42000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:38.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.079+0000 7fc42cff9700 1 -- 172.21.15.152:0/2891427266 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc420009320 con 0x7fc4300f6510 2023-12-06T03:58:38.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.079+0000 7fc4369d3700 1 -- 172.21.15.152:0/2891427266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc430106e20 con 0x7fc4300f6510 2023-12-06T03:58:38.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.079+0000 7fc42cff9700 1 -- 172.21.15.152:0/2891427266 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc420013f00 con 0x7fc4300f6510 2023-12-06T03:58:38.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.079+0000 7fc42cff9700 1 -- 172.21.15.152:0/2891427266 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc420014d90 con 0x7fc4300f6510 2023-12-06T03:58:38.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.079+0000 7fc4369d3700 1 -- 172.21.15.152:0/2891427266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4301072b0 con 0x7fc4300f6510 2023-12-06T03:58:38.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.080+0000 7fc42cff9700 1 -- 172.21.15.152:0/2891427266 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc420014420 con 0x7fc4300f6510 2023-12-06T03:58:38.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.080+0000 7fc4369d3700 1 -- 172.21.15.152:0/2891427266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc430045ad0 con 0x7fc4300f6510 2023-12-06T03:58:38.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.080+0000 7fc42cff9700 1 --2- 172.21.15.152:0/2891427266 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc41802ff60 0x7fc418032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:38.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.080+0000 7fc42cff9700 1 -- 172.21.15.152:0/2891427266 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4200410c0 con 0x7fc4300f6510 2023-12-06T03:58:38.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.081+0000 7fc42f7fe700 1 --2- 172.21.15.152:0/2891427266 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc41802ff60 0x7fc418032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:38.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.081+0000 7fc42f7fe700 1 --2- 172.21.15.152:0/2891427266 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc41802ff60 0x7fc418032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fc4240096a0 tx=0x7fc424006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:38.086 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.085+0000 7fc42cff9700 1 -- 172.21.15.152:0/2891427266 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4200095f0 con 0x7fc4300f6510 2023-12-06T03:58:38.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:36.437936+0000 mgr.smithi132.gokxja (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:38.292 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.290+0000 7fc4369d3700 1 -- 172.21.15.152:0/2891427266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4300fbf60 con 0x7fc4300f6510 2023-12-06T03:58:38.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.291+0000 7fc42cff9700 1 -- 172.21.15.152:0/2891427266 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc42001db70 con 0x7fc4300f6510 2023-12-06T03:58:38.293 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:38.293 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:38.294 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.293+0000 7fc4369d3700 1 -- 172.21.15.152:0/2891427266 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc41802ff60 msgr2=0x7fc418032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:38.294 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.293+0000 7fc4369d3700 1 --2- 172.21.15.152:0/2891427266 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc41802ff60 0x7fc418032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fc4240096a0 tx=0x7fc424006b40).stop 2023-12-06T03:58:38.294 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.293+0000 7fc4369d3700 1 -- 172.21.15.152:0/2891427266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f6510 msgr2=0x7fc430106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:38.295 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.293+0000 7fc4369d3700 1 --2- 172.21.15.152:0/2891427266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f6510 0x7fc430106730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fc420013510 tx=0x7fc42000b040).stop 2023-12-06T03:58:38.295 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.293+0000 7fc4369d3700 1 -- 172.21.15.152:0/2891427266 shutdown_connections 2023-12-06T03:58:38.295 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.293+0000 7fc4369d3700 1 --2- 172.21.15.152:0/2891427266 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc41802ff60 0x7fc418032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:38.295 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.293+0000 7fc4369d3700 1 --2- 172.21.15.152:0/2891427266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc4300f6510 0x7fc430106730 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:38.295 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.293+0000 7fc4369d3700 1 -- 172.21.15.152:0/2891427266 >> 172.21.15.152:0/2891427266 conn(0x7fc4300f1c70 msgr2=0x7fc4300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:38.295 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.293+0000 7fc4369d3700 1 -- 172.21.15.152:0/2891427266 shutdown_connections 2023-12-06T03:58:38.295 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:38.293+0000 7fc4369d3700 1 -- 172.21.15.152:0/2891427266 wait complete. 2023-12-06T03:58:38.295 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:39.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:38 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:38.292371+0000 mon.smithi132 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.152:0/2891427266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:39.661 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:39.662 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:40.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:39 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:38.438201+0000 mgr.smithi132.gokxja (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:40.890 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.886+0000 7f35c32f0700 1 -- 172.21.15.152:0/3250869024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35bc0f6510 msgr2=0x7f35bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:40.890 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.886+0000 7f35c32f0700 1 --2- 172.21.15.152:0/3250869024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35bc0f6510 0x7f35bc0f6930 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f35ac009fb0 tx=0x7f35ac009b30).stop 2023-12-06T03:58:40.890 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.886+0000 7f35c32f0700 1 -- 172.21.15.152:0/3250869024 shutdown_connections 2023-12-06T03:58:40.890 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.886+0000 7f35c32f0700 1 --2- 172.21.15.152:0/3250869024 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35bc0f6510 0x7f35bc0f6930 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:40.890 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.886+0000 7f35c32f0700 1 -- 172.21.15.152:0/3250869024 >> 172.21.15.152:0/3250869024 conn(0x7f35bc0f1c70 msgr2=0x7f35bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:40.890 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.886+0000 7f35c32f0700 1 -- 172.21.15.152:0/3250869024 shutdown_connections 2023-12-06T03:58:40.890 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.886+0000 7f35c32f0700 1 -- 172.21.15.152:0/3250869024 wait complete. 2023-12-06T03:58:40.890 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.887+0000 7f35c32f0700 1 Processor -- start 2023-12-06T03:58:40.890 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.887+0000 7f35c32f0700 1 -- start start 2023-12-06T03:58:40.890 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.887+0000 7f35c32f0700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35bc0f6510 0x7f35bc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:40.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.887+0000 7f35c32f0700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35bc069e20 con 0x7f35bc0f6510 2023-12-06T03:58:40.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.887+0000 7f35c108c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35bc0f6510 0x7f35bc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:40.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.887+0000 7f35c108c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35bc0f6510 0x7f35bc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:39964/0 (socket says 172.21.15.152:39964) 2023-12-06T03:58:40.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.887+0000 7f35c108c700 1 -- 172.21.15.152:0/1810622799 learned_addr learned my addr 172.21.15.152:0/1810622799 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:40.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.887+0000 7f35c108c700 1 -- 172.21.15.152:0/1810622799 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35ac005040 con 0x7f35bc0f6510 2023-12-06T03:58:40.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.888+0000 7f35c108c700 1 --2- 172.21.15.152:0/1810622799 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35bc0f6510 0x7f35bc06b9a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f35ac013510 tx=0x7f35ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:40.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.888+0000 7f35b1ffb700 1 -- 172.21.15.152:0/1810622799 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35ac009320 con 0x7f35bc0f6510 2023-12-06T03:58:40.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.888+0000 7f35c32f0700 1 -- 172.21.15.152:0/1810622799 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35bc069fc0 con 0x7f35bc0f6510 2023-12-06T03:58:40.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.888+0000 7f35b1ffb700 1 -- 172.21.15.152:0/1810622799 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f35ac013f00 con 0x7f35bc0f6510 2023-12-06T03:58:40.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.888+0000 7f35b1ffb700 1 -- 172.21.15.152:0/1810622799 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f35ac014d90 con 0x7f35bc0f6510 2023-12-06T03:58:40.891 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.888+0000 7f35c32f0700 1 -- 172.21.15.152:0/1810622799 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35bc06a450 con 0x7f35bc0f6510 2023-12-06T03:58:40.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.889+0000 7f35b1ffb700 1 -- 172.21.15.152:0/1810622799 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f35ac0095f0 con 0x7f35bc0f6510 2023-12-06T03:58:40.892 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.889+0000 7f35c32f0700 1 -- 172.21.15.152:0/1810622799 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35bc059100 con 0x7f35bc0f6510 2023-12-06T03:58:40.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.889+0000 7f35b1ffb700 1 --2- 172.21.15.152:0/1810622799 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f35a802ff60 0x7f35a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:40.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.889+0000 7f35b1ffb700 1 -- 172.21.15.152:0/1810622799 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f35ac0410c0 con 0x7f35bc0f6510 2023-12-06T03:58:40.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.889+0000 7f35c088b700 1 --2- 172.21.15.152:0/1810622799 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f35a802ff60 0x7f35a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:40.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.890+0000 7f35c088b700 1 --2- 172.21.15.152:0/1810622799 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f35a802ff60 0x7f35a8032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f35b80096a0 tx=0x7f35b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:40.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:40.894+0000 7f35b1ffb700 1 -- 172.21.15.152:0/1810622799 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f35ac014420 con 0x7f35bc0f6510 2023-12-06T03:58:41.101 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:41.099+0000 7f35c32f0700 1 -- 172.21.15.152:0/1810622799 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f35bc0fbf60 con 0x7f35bc0f6510 2023-12-06T03:58:41.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:41.099+0000 7f35b1ffb700 1 -- 172.21.15.152:0/1810622799 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f35ac01d7f0 con 0x7f35bc0f6510 2023-12-06T03:58:41.102 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:41.103 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:41.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:41.101+0000 7f35c32f0700 1 -- 172.21.15.152:0/1810622799 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f35a802ff60 msgr2=0x7f35a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:41.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:41.101+0000 7f35c32f0700 1 --2- 172.21.15.152:0/1810622799 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f35a802ff60 0x7f35a8032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f35b80096a0 tx=0x7f35b8006b40).stop 2023-12-06T03:58:41.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:41.101+0000 7f35c32f0700 1 -- 172.21.15.152:0/1810622799 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35bc0f6510 msgr2=0x7f35bc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:41.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:41.101+0000 7f35c32f0700 1 --2- 172.21.15.152:0/1810622799 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35bc0f6510 0x7f35bc06b9a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f35ac013510 tx=0x7f35ac00b040).stop 2023-12-06T03:58:41.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:41.101+0000 7f35c32f0700 1 -- 172.21.15.152:0/1810622799 shutdown_connections 2023-12-06T03:58:41.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:41.101+0000 7f35c32f0700 1 --2- 172.21.15.152:0/1810622799 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f35a802ff60 0x7f35a8032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:41.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:41.101+0000 7f35c32f0700 1 --2- 172.21.15.152:0/1810622799 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f35bc0f6510 0x7f35bc06b9a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:41.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:41.101+0000 7f35c32f0700 1 -- 172.21.15.152:0/1810622799 >> 172.21.15.152:0/1810622799 conn(0x7f35bc0f1c70 msgr2=0x7f35bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:41.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:41.101+0000 7f35c32f0700 1 -- 172.21.15.152:0/1810622799 shutdown_connections 2023-12-06T03:58:41.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:41.101+0000 7f35c32f0700 1 -- 172.21.15.152:0/1810622799 wait complete. 2023-12-06T03:58:41.104 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:42.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:41 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:40.438502+0000 mgr.smithi132.gokxja (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:42.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:41 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:41.100985+0000 mon.smithi132 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.152:0/1810622799' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:42.486 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:42.487 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:43.586 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.584+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2072790992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c0c0f5410 msgr2=0x7f1c0c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:43.587 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.584+0000 7f1c11fc9700 1 --2- 172.21.15.152:0/2072790992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c0f5830 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f1bf4009fb0 tx=0x7f1bf4009b30).stop 2023-12-06T03:58:43.587 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.585+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2072790992 shutdown_connections 2023-12-06T03:58:43.587 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.585+0000 7f1c11fc9700 1 --2- 172.21.15.152:0/2072790992 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c0f5830 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:43.587 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.585+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2072790992 >> 172.21.15.152:0/2072790992 conn(0x7f1c0c0f0b70 msgr2=0x7f1c0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:43.587 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.585+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2072790992 shutdown_connections 2023-12-06T03:58:43.587 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.585+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2072790992 wait complete. 2023-12-06T03:58:43.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.585+0000 7f1c11fc9700 1 Processor -- start 2023-12-06T03:58:43.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.585+0000 7f1c11fc9700 1 -- start start 2023-12-06T03:58:43.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.585+0000 7f1c11fc9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:43.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.585+0000 7f1c11fc9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0c106c40 con 0x7f1c0c0f5410 2023-12-06T03:58:43.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.586+0000 7f1c0b7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:43.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.586+0000 7f1c0b7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:39976/0 (socket says 172.21.15.152:39976) 2023-12-06T03:58:43.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.586+0000 7f1c0b7fe700 1 -- 172.21.15.152:0/2023719969 learned_addr learned my addr 172.21.15.152:0/2023719969 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:43.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.586+0000 7f1c0b7fe700 1 -- 172.21.15.152:0/2023719969 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf4005040 con 0x7f1c0c0f5410 2023-12-06T03:58:43.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.586+0000 7f1c0b7fe700 1 --2- 172.21.15.152:0/2023719969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c106750 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f1bf4009f80 tx=0x7f1bf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:43.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.586+0000 7f1c03fff700 1 -- 172.21.15.152:0/2023719969 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bf4009320 con 0x7f1c0c0f5410 2023-12-06T03:58:43.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.586+0000 7f1c03fff700 1 -- 172.21.15.152:0/2023719969 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bf4013ec0 con 0x7f1c0c0f5410 2023-12-06T03:58:43.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.586+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2023719969 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0c106e40 con 0x7f1c0c0f5410 2023-12-06T03:58:43.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.586+0000 7f1c03fff700 1 -- 172.21.15.152:0/2023719969 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bf401ece0 con 0x7f1c0c0f5410 2023-12-06T03:58:43.589 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.586+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2023719969 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0c1072d0 con 0x7f1c0c0f5410 2023-12-06T03:58:43.590 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.587+0000 7f1c03fff700 1 -- 172.21.15.152:0/2023719969 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1bf401e420 con 0x7f1c0c0f5410 2023-12-06T03:58:43.590 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.587+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2023719969 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c0c045ad0 con 0x7f1c0c0f5410 2023-12-06T03:58:43.590 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.587+0000 7f1c03fff700 1 --2- 172.21.15.152:0/2023719969 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1bf802ff60 0x7f1bf8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:43.590 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.587+0000 7f1c03fff700 1 -- 172.21.15.152:0/2023719969 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1bf4040e30 con 0x7f1c0c0f5410 2023-12-06T03:58:43.590 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.587+0000 7f1c0affd700 1 --2- 172.21.15.152:0/2023719969 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1bf802ff60 0x7f1bf8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:43.590 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.588+0000 7f1c0affd700 1 --2- 172.21.15.152:0/2023719969 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1bf802ff60 0x7f1bf8032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f1bfc0096a0 tx=0x7f1bfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:43.593 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.592+0000 7f1c03fff700 1 -- 172.21.15.152:0/2023719969 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1bf401e730 con 0x7f1c0c0f5410 2023-12-06T03:58:43.798 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.796+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2023719969 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c0c108180 con 0x7f1c0c0f5410 2023-12-06T03:58:43.798 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.797+0000 7f1c03fff700 1 -- 172.21.15.152:0/2023719969 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1bf402c2d0 con 0x7f1c0c0f5410 2023-12-06T03:58:43.799 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:43.799 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:43.800 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.798+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2023719969 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1bf802ff60 msgr2=0x7f1bf8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:43.800 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.798+0000 7f1c11fc9700 1 --2- 172.21.15.152:0/2023719969 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1bf802ff60 0x7f1bf8032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f1bfc0096a0 tx=0x7f1bfc006b40).stop 2023-12-06T03:58:43.800 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.798+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2023719969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c0c0f5410 msgr2=0x7f1c0c106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:43.800 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.798+0000 7f1c11fc9700 1 --2- 172.21.15.152:0/2023719969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c106750 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f1bf4009f80 tx=0x7f1bf400b040).stop 2023-12-06T03:58:43.800 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.798+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2023719969 shutdown_connections 2023-12-06T03:58:43.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.798+0000 7f1c11fc9700 1 --2- 172.21.15.152:0/2023719969 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1bf802ff60 0x7f1bf8032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:43.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.798+0000 7f1c11fc9700 1 --2- 172.21.15.152:0/2023719969 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1c0c0f5410 0x7f1c0c106750 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:43.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.798+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2023719969 >> 172.21.15.152:0/2023719969 conn(0x7f1c0c0f0b70 msgr2=0x7f1c0c05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:43.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.798+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2023719969 shutdown_connections 2023-12-06T03:58:43.801 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:43.799+0000 7f1c11fc9700 1 -- 172.21.15.152:0/2023719969 wait complete. 2023-12-06T03:58:43.801 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:44.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:42.438771+0000 mgr.smithi132.gokxja (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:45.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:44 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:43.798228+0000 mon.smithi132 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.152:0/2023719969' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:45.185 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:45.185 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:46.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:45 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:44.439096+0000 mgr.smithi132.gokxja (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:46.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.337+0000 7f91539b0700 1 -- 172.21.15.152:0/2737504608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f914c0f5950 msgr2=0x7f914c0f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:46.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.337+0000 7f91539b0700 1 --2- 172.21.15.152:0/2737504608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f914c0f5950 0x7f914c0f5d70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f913c009fb0 tx=0x7f913c009b30).stop 2023-12-06T03:58:46.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.337+0000 7f91539b0700 1 -- 172.21.15.152:0/2737504608 shutdown_connections 2023-12-06T03:58:46.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.337+0000 7f91539b0700 1 --2- 172.21.15.152:0/2737504608 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f914c0f5950 0x7f914c0f5d70 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:46.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.337+0000 7f91539b0700 1 -- 172.21.15.152:0/2737504608 >> 172.21.15.152:0/2737504608 conn(0x7f914c0f0260 msgr2=0x7f914c0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:46.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.337+0000 7f91539b0700 1 -- 172.21.15.152:0/2737504608 shutdown_connections 2023-12-06T03:58:46.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.337+0000 7f91539b0700 1 -- 172.21.15.152:0/2737504608 wait complete. 2023-12-06T03:58:46.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.338+0000 7f91539b0700 1 Processor -- start 2023-12-06T03:58:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.338+0000 7f91539b0700 1 -- start start 2023-12-06T03:58:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.338+0000 7f91539b0700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f914c0f5950 0x7f914c1093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.338+0000 7f91539b0700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f914c0f5370 con 0x7f914c0f5950 2023-12-06T03:58:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.338+0000 7f915174c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f914c0f5950 0x7f914c1093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.338+0000 7f915174c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f914c0f5950 0x7f914c1093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:39988/0 (socket says 172.21.15.152:39988) 2023-12-06T03:58:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.338+0000 7f915174c700 1 -- 172.21.15.152:0/3875340842 learned_addr learned my addr 172.21.15.152:0/3875340842 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.338+0000 7f915174c700 1 -- 172.21.15.152:0/3875340842 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f913c005040 con 0x7f914c0f5950 2023-12-06T03:58:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.339+0000 7f915174c700 1 --2- 172.21.15.152:0/3875340842 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f914c0f5950 0x7f914c1093d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f913c014000 tx=0x7f913c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.339+0000 7f9141ffb700 1 -- 172.21.15.152:0/3875340842 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f913c0095f0 con 0x7f914c0f5950 2023-12-06T03:58:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.339+0000 7f91539b0700 1 -- 172.21.15.152:0/3875340842 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f914c0f5570 con 0x7f914c0f5950 2023-12-06T03:58:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.339+0000 7f9141ffb700 1 -- 172.21.15.152:0/3875340842 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f913c007e70 con 0x7f914c0f5950 2023-12-06T03:58:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.339+0000 7f9141ffb700 1 -- 172.21.15.152:0/3875340842 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f913c014ea0 con 0x7f914c0f5950 2023-12-06T03:58:46.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.339+0000 7f91539b0700 1 -- 172.21.15.152:0/3875340842 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f914c109b00 con 0x7f914c0f5950 2023-12-06T03:58:46.343 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.340+0000 7f9141ffb700 1 -- 172.21.15.152:0/3875340842 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f913c025070 con 0x7f914c0f5950 2023-12-06T03:58:46.343 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.340+0000 7f91539b0700 1 -- 172.21.15.152:0/3875340842 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f914c045ad0 con 0x7f914c0f5950 2023-12-06T03:58:46.343 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.340+0000 7f9141ffb700 1 --2- 172.21.15.152:0/3875340842 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f913802ff60 0x7f9138032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:46.343 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.340+0000 7f9141ffb700 1 -- 172.21.15.152:0/3875340842 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f913c040a30 con 0x7f914c0f5950 2023-12-06T03:58:46.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.340+0000 7f9150f4b700 1 --2- 172.21.15.152:0/3875340842 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f913802ff60 0x7f9138032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:46.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.341+0000 7f9150f4b700 1 --2- 172.21.15.152:0/3875340842 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f913802ff60 0x7f9138032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f91480096a0 tx=0x7f9148006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:46.346 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.345+0000 7f9141ffb700 1 -- 172.21.15.152:0/3875340842 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f913c0079c0 con 0x7f914c0f5950 2023-12-06T03:58:46.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.548+0000 7f91539b0700 1 -- 172.21.15.152:0/3875340842 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f914c0fae40 con 0x7f914c0f5950 2023-12-06T03:58:46.694 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.548+0000 7f9141ffb700 1 -- 172.21.15.152:0/3875340842 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f913c00f300 con 0x7f914c0f5950 2023-12-06T03:58:46.696 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:46.696 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:46.698 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.550+0000 7f91539b0700 1 -- 172.21.15.152:0/3875340842 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f913802ff60 msgr2=0x7f9138032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:46.698 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.550+0000 7f91539b0700 1 --2- 172.21.15.152:0/3875340842 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f913802ff60 0x7f9138032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f91480096a0 tx=0x7f9148006b40).stop 2023-12-06T03:58:46.698 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.550+0000 7f91539b0700 1 -- 172.21.15.152:0/3875340842 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f914c0f5950 msgr2=0x7f914c1093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:46.699 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.550+0000 7f91539b0700 1 --2- 172.21.15.152:0/3875340842 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f914c0f5950 0x7f914c1093d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f913c014000 tx=0x7f913c00b040).stop 2023-12-06T03:58:46.699 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.550+0000 7f91539b0700 1 -- 172.21.15.152:0/3875340842 shutdown_connections 2023-12-06T03:58:46.699 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.550+0000 7f91539b0700 1 --2- 172.21.15.152:0/3875340842 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f913802ff60 0x7f9138032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:46.699 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.550+0000 7f91539b0700 1 --2- 172.21.15.152:0/3875340842 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f914c0f5950 0x7f914c1093d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:46.700 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.550+0000 7f91539b0700 1 -- 172.21.15.152:0/3875340842 >> 172.21.15.152:0/3875340842 conn(0x7f914c0f0260 msgr2=0x7f914c0f0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:46.700 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.550+0000 7f91539b0700 1 -- 172.21.15.152:0/3875340842 shutdown_connections 2023-12-06T03:58:46.700 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:46.550+0000 7f91539b0700 1 -- 172.21.15.152:0/3875340842 wait complete. 2023-12-06T03:58:46.700 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:47.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:47 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:46.439379+0000 mgr.smithi132.gokxja (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:47.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:47 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:46.549888+0000 mon.smithi132 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.152:0/3875340842' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:47.926 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:47.926 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:49.018 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.016+0000 7f664a7fa700 1 -- 172.21.15.152:0/1871269476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6644060280 msgr2=0x7f66440f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:49.018 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.016+0000 7f664a7fa700 1 --2- 172.21.15.152:0/1871269476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6644060280 0x7f66440f90f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f6634009fb0 tx=0x7f6634009b30).stop 2023-12-06T03:58:49.019 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.016+0000 7f664a7fa700 1 -- 172.21.15.152:0/1871269476 shutdown_connections 2023-12-06T03:58:49.019 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.016+0000 7f664a7fa700 1 --2- 172.21.15.152:0/1871269476 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6644060280 0x7f66440f90f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:49.019 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.016+0000 7f664a7fa700 1 -- 172.21.15.152:0/1871269476 >> 172.21.15.152:0/1871269476 conn(0x7f66440f0b70 msgr2=0x7f66440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:49.019 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.016+0000 7f664a7fa700 1 -- 172.21.15.152:0/1871269476 shutdown_connections 2023-12-06T03:58:49.019 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.016+0000 7f664a7fa700 1 -- 172.21.15.152:0/1871269476 wait complete. 2023-12-06T03:58:49.019 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.017+0000 7f664a7fa700 1 Processor -- start 2023-12-06T03:58:49.021 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.017+0000 7f664a7fa700 1 -- start start 2023-12-06T03:58:49.021 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.017+0000 7f664a7fa700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6644060280 0x7f66441066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:49.021 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.017+0000 7f664a7fa700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6644106bd0 con 0x7f6644060280 2023-12-06T03:58:49.021 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.017+0000 7f6643fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6644060280 0x7f66441066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:49.021 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.017+0000 7f6643fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6644060280 0x7f66441066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40006/0 (socket says 172.21.15.152:40006) 2023-12-06T03:58:49.021 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.017+0000 7f6643fff700 1 -- 172.21.15.152:0/3277625048 learned_addr learned my addr 172.21.15.152:0/3277625048 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:49.021 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.018+0000 7f6643fff700 1 -- 172.21.15.152:0/3277625048 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6634005040 con 0x7f6644060280 2023-12-06T03:58:49.021 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.018+0000 7f6643fff700 1 --2- 172.21.15.152:0/3277625048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6644060280 0x7f66441066e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f6634013510 tx=0x7f663400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:49.021 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.018+0000 7f6640ff9700 1 -- 172.21.15.152:0/3277625048 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6634009320 con 0x7f6644060280 2023-12-06T03:58:49.021 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.018+0000 7f6640ff9700 1 -- 172.21.15.152:0/3277625048 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6634013f00 con 0x7f6644060280 2023-12-06T03:58:49.021 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.018+0000 7f664a7fa700 1 -- 172.21.15.152:0/3277625048 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6644106dd0 con 0x7f6644060280 2023-12-06T03:58:49.021 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.018+0000 7f6640ff9700 1 -- 172.21.15.152:0/3277625048 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6634014dc0 con 0x7f6644060280 2023-12-06T03:58:49.021 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.018+0000 7f664a7fa700 1 -- 172.21.15.152:0/3277625048 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6644107260 con 0x7f6644060280 2023-12-06T03:58:49.021 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.019+0000 7f6640ff9700 1 -- 172.21.15.152:0/3277625048 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f663401d3d0 con 0x7f6644060280 2023-12-06T03:58:49.022 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.019+0000 7f6640ff9700 1 --2- 172.21.15.152:0/3277625048 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f662c02ff60 0x7f662c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:49.022 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.019+0000 7f664a7fa700 1 -- 172.21.15.152:0/3277625048 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6644045ad0 con 0x7f6644060280 2023-12-06T03:58:49.022 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.019+0000 7f6640ff9700 1 -- 172.21.15.152:0/3277625048 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6634041100 con 0x7f6644060280 2023-12-06T03:58:49.023 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.019+0000 7f66437fe700 1 --2- 172.21.15.152:0/3277625048 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f662c02ff60 0x7f662c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:49.023 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.020+0000 7f66437fe700 1 --2- 172.21.15.152:0/3277625048 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f662c02ff60 0x7f662c032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f66380096a0 tx=0x7f6638006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:49.025 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.024+0000 7f6640ff9700 1 -- 172.21.15.152:0/3277625048 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6634012900 con 0x7f6644060280 2023-12-06T03:58:49.230 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.228+0000 7f664a7fa700 1 -- 172.21.15.152:0/3277625048 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6644108110 con 0x7f6644060280 2023-12-06T03:58:49.231 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.229+0000 7f6640ff9700 1 -- 172.21.15.152:0/3277625048 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f663402b020 con 0x7f6644060280 2023-12-06T03:58:49.231 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:49.231 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:49.232 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.231+0000 7f664a7fa700 1 -- 172.21.15.152:0/3277625048 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f662c02ff60 msgr2=0x7f662c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:49.232 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.231+0000 7f664a7fa700 1 --2- 172.21.15.152:0/3277625048 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f662c02ff60 0x7f662c032420 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f66380096a0 tx=0x7f6638006b40).stop 2023-12-06T03:58:49.232 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.231+0000 7f664a7fa700 1 -- 172.21.15.152:0/3277625048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6644060280 msgr2=0x7f66441066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:49.233 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.231+0000 7f664a7fa700 1 --2- 172.21.15.152:0/3277625048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6644060280 0x7f66441066e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f6634013510 tx=0x7f663400b040).stop 2023-12-06T03:58:49.233 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.231+0000 7f664a7fa700 1 -- 172.21.15.152:0/3277625048 shutdown_connections 2023-12-06T03:58:49.233 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.231+0000 7f664a7fa700 1 --2- 172.21.15.152:0/3277625048 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f662c02ff60 0x7f662c032420 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:49.233 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.231+0000 7f664a7fa700 1 --2- 172.21.15.152:0/3277625048 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6644060280 0x7f66441066e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:49.233 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.231+0000 7f664a7fa700 1 -- 172.21.15.152:0/3277625048 >> 172.21.15.152:0/3277625048 conn(0x7f66440f0b70 msgr2=0x7f66440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:49.233 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.231+0000 7f664a7fa700 1 -- 172.21.15.152:0/3277625048 shutdown_connections 2023-12-06T03:58:49.233 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:49.231+0000 7f664a7fa700 1 -- 172.21.15.152:0/3277625048 wait complete. 2023-12-06T03:58:49.233 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:49.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:49 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:48.439647+0000 mgr.smithi132.gokxja (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:49.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:49 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:49.230536+0000 mon.smithi132 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.152:0/3277625048' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:50.605 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:50.605 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:51.723 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.718+0000 7f76dd403700 1 -- 172.21.15.152:0/1332092739 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76d80f51c0 msgr2=0x7f76d80f55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:51.723 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.718+0000 7f76dd403700 1 --2- 172.21.15.152:0/1332092739 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76d80f51c0 0x7f76d80f55e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f76c8009fb0 tx=0x7f76c8009b30).stop 2023-12-06T03:58:51.723 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.718+0000 7f76dd403700 1 -- 172.21.15.152:0/1332092739 shutdown_connections 2023-12-06T03:58:51.723 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.718+0000 7f76dd403700 1 --2- 172.21.15.152:0/1332092739 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76d80f51c0 0x7f76d80f55e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:51.723 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.718+0000 7f76dd403700 1 -- 172.21.15.152:0/1332092739 >> 172.21.15.152:0/1332092739 conn(0x7f76d80f0920 msgr2=0x7f76d80f2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:51.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.718+0000 7f76dd403700 1 -- 172.21.15.152:0/1332092739 shutdown_connections 2023-12-06T03:58:51.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.718+0000 7f76dd403700 1 -- 172.21.15.152:0/1332092739 wait complete. 2023-12-06T03:58:51.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.718+0000 7f76dd403700 1 Processor -- start 2023-12-06T03:58:51.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.718+0000 7f76dd403700 1 -- start start 2023-12-06T03:58:51.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.718+0000 7f76dd403700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76d80f51c0 0x7f76d80fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:51.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.718+0000 7f76dd403700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76d80fb640 con 0x7f76d80f51c0 2023-12-06T03:58:51.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.719+0000 7f76d6ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76d80f51c0 0x7f76d80fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:51.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.719+0000 7f76d6ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76d80f51c0 0x7f76d80fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40116/0 (socket says 172.21.15.152:40116) 2023-12-06T03:58:51.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.719+0000 7f76d6ffd700 1 -- 172.21.15.152:0/4201599120 learned_addr learned my addr 172.21.15.152:0/4201599120 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:51.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.719+0000 7f76d6ffd700 1 -- 172.21.15.152:0/4201599120 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c8005040 con 0x7f76d80f51c0 2023-12-06T03:58:51.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.719+0000 7f76d6ffd700 1 --2- 172.21.15.152:0/4201599120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76d80f51c0 0x7f76d80fd1c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f76c8013510 tx=0x7f76c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:51.724 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.719+0000 7f76bf7fe700 1 -- 172.21.15.152:0/4201599120 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76c8009320 con 0x7f76d80f51c0 2023-12-06T03:58:51.725 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.719+0000 7f76dd403700 1 -- 172.21.15.152:0/4201599120 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76d80fb7e0 con 0x7f76d80f51c0 2023-12-06T03:58:51.725 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.719+0000 7f76bf7fe700 1 -- 172.21.15.152:0/4201599120 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76c8013f00 con 0x7f76d80f51c0 2023-12-06T03:58:51.725 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.719+0000 7f76bf7fe700 1 -- 172.21.15.152:0/4201599120 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76c8014d90 con 0x7f76d80f51c0 2023-12-06T03:58:51.725 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.719+0000 7f76dd403700 1 -- 172.21.15.152:0/4201599120 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76d80fbc70 con 0x7f76d80f51c0 2023-12-06T03:58:51.725 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.720+0000 7f76bf7fe700 1 -- 172.21.15.152:0/4201599120 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76c80095f0 con 0x7f76d80f51c0 2023-12-06T03:58:51.725 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.720+0000 7f76dd403700 1 -- 172.21.15.152:0/4201599120 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76d8059100 con 0x7f76d80f51c0 2023-12-06T03:58:51.725 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.720+0000 7f76bf7fe700 1 --2- 172.21.15.152:0/4201599120 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f76c002ff60 0x7f76c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:51.725 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.720+0000 7f76bf7fe700 1 -- 172.21.15.152:0/4201599120 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76c80410c0 con 0x7f76d80f51c0 2023-12-06T03:58:51.725 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.721+0000 7f76d67fc700 1 --2- 172.21.15.152:0/4201599120 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f76c002ff60 0x7f76c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:51.725 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.721+0000 7f76d67fc700 1 --2- 172.21.15.152:0/4201599120 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f76c002ff60 0x7f76c0032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f76cc0096a0 tx=0x7f76cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:51.727 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.725+0000 7f76bf7fe700 1 -- 172.21.15.152:0/4201599120 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76c8014420 con 0x7f76d80f51c0 2023-12-06T03:58:51.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:51 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:50.439991+0000 mgr.smithi132.gokxja (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:51.931 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.929+0000 7f76dd403700 1 -- 172.21.15.152:0/4201599120 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76d8045ad0 con 0x7f76d80f51c0 2023-12-06T03:58:51.932 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.930+0000 7f76bf7fe700 1 -- 172.21.15.152:0/4201599120 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f76c801d7f0 con 0x7f76d80f51c0 2023-12-06T03:58:51.932 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:51.932 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:51.933 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.932+0000 7f76dd403700 1 -- 172.21.15.152:0/4201599120 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f76c002ff60 msgr2=0x7f76c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:51.933 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.932+0000 7f76dd403700 1 --2- 172.21.15.152:0/4201599120 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f76c002ff60 0x7f76c0032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f76cc0096a0 tx=0x7f76cc006b40).stop 2023-12-06T03:58:51.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.932+0000 7f76dd403700 1 -- 172.21.15.152:0/4201599120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76d80f51c0 msgr2=0x7f76d80fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:51.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.932+0000 7f76dd403700 1 --2- 172.21.15.152:0/4201599120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76d80f51c0 0x7f76d80fd1c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f76c8013510 tx=0x7f76c800b040).stop 2023-12-06T03:58:51.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.932+0000 7f76dd403700 1 -- 172.21.15.152:0/4201599120 shutdown_connections 2023-12-06T03:58:51.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.932+0000 7f76dd403700 1 --2- 172.21.15.152:0/4201599120 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f76c002ff60 0x7f76c0032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:51.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.932+0000 7f76dd403700 1 --2- 172.21.15.152:0/4201599120 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f76d80f51c0 0x7f76d80fd1c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:51.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.932+0000 7f76dd403700 1 -- 172.21.15.152:0/4201599120 >> 172.21.15.152:0/4201599120 conn(0x7f76d80f0920 msgr2=0x7f76d805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:51.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.932+0000 7f76dd403700 1 -- 172.21.15.152:0/4201599120 shutdown_connections 2023-12-06T03:58:51.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:51.932+0000 7f76dd403700 1 -- 172.21.15.152:0/4201599120 wait complete. 2023-12-06T03:58:51.934 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:52.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:52 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:51.931552+0000 mon.smithi132 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.152:0/4201599120' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:53.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:53.323 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:53.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:53 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:52.440389+0000 mgr.smithi132.gokxja (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:54.471 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.466+0000 7f4896aa5700 1 -- 172.21.15.152:0/4125993020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48900f5410 msgr2=0x7f48900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:54.471 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.466+0000 7f4896aa5700 1 --2- 172.21.15.152:0/4125993020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48900f5410 0x7f48900f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f4884009fb0 tx=0x7f4884009b30).stop 2023-12-06T03:58:54.471 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.466+0000 7f4896aa5700 1 -- 172.21.15.152:0/4125993020 shutdown_connections 2023-12-06T03:58:54.471 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.466+0000 7f4896aa5700 1 --2- 172.21.15.152:0/4125993020 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48900f5410 0x7f48900f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:54.471 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.466+0000 7f4896aa5700 1 -- 172.21.15.152:0/4125993020 >> 172.21.15.152:0/4125993020 conn(0x7f48900f0b70 msgr2=0x7f48900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:54.472 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.467+0000 7f4896aa5700 1 -- 172.21.15.152:0/4125993020 shutdown_connections 2023-12-06T03:58:54.472 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.467+0000 7f4896aa5700 1 -- 172.21.15.152:0/4125993020 wait complete. 2023-12-06T03:58:54.472 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.467+0000 7f4896aa5700 1 Processor -- start 2023-12-06T03:58:54.472 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.467+0000 7f4896aa5700 1 -- start start 2023-12-06T03:58:54.472 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.467+0000 7f4896aa5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48900f5410 0x7f4890109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:54.472 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.467+0000 7f4896aa5700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4890060220 con 0x7f48900f5410 2023-12-06T03:58:54.472 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.468+0000 7f4894841700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48900f5410 0x7f4890109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:54.472 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.468+0000 7f4894841700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48900f5410 0x7f4890109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40132/0 (socket says 172.21.15.152:40132) 2023-12-06T03:58:54.472 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.468+0000 7f4894841700 1 -- 172.21.15.152:0/4266735313 learned_addr learned my addr 172.21.15.152:0/4266735313 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:54.472 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.468+0000 7f4894841700 1 -- 172.21.15.152:0/4266735313 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4884005040 con 0x7f48900f5410 2023-12-06T03:58:54.472 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.468+0000 7f4894841700 1 --2- 172.21.15.152:0/4266735313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48900f5410 0x7f4890109450 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f4884013510 tx=0x7f488400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:54.472 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.468+0000 7f488d7fa700 1 -- 172.21.15.152:0/4266735313 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4884009320 con 0x7f48900f5410 2023-12-06T03:58:54.472 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.468+0000 7f488d7fa700 1 -- 172.21.15.152:0/4266735313 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4884013f00 con 0x7f48900f5410 2023-12-06T03:58:54.472 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.468+0000 7f4896aa5700 1 -- 172.21.15.152:0/4266735313 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4890060420 con 0x7f48900f5410 2023-12-06T03:58:54.473 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.468+0000 7f488d7fa700 1 -- 172.21.15.152:0/4266735313 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4884014d90 con 0x7f48900f5410 2023-12-06T03:58:54.473 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.468+0000 7f4896aa5700 1 -- 172.21.15.152:0/4266735313 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4890109b00 con 0x7f48900f5410 2023-12-06T03:58:54.473 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.469+0000 7f488d7fa700 1 -- 172.21.15.152:0/4266735313 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4884014420 con 0x7f48900f5410 2023-12-06T03:58:54.473 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.469+0000 7f4896aa5700 1 -- 172.21.15.152:0/4266735313 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4890059100 con 0x7f48900f5410 2023-12-06T03:58:54.473 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.469+0000 7f488d7fa700 1 --2- 172.21.15.152:0/4266735313 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f488002ff60 0x7f4880032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:54.473 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.469+0000 7f488d7fa700 1 -- 172.21.15.152:0/4266735313 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48840410c0 con 0x7f48900f5410 2023-12-06T03:58:54.473 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.469+0000 7f488ffff700 1 --2- 172.21.15.152:0/4266735313 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f488002ff60 0x7f4880032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:54.473 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.470+0000 7f488ffff700 1 --2- 172.21.15.152:0/4266735313 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f488002ff60 0x7f4880032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f487c0096a0 tx=0x7f487c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:54.475 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.474+0000 7f488d7fa700 1 -- 172.21.15.152:0/4266735313 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f48840095f0 con 0x7f48900f5410 2023-12-06T03:58:54.680 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.679+0000 7f4896aa5700 1 -- 172.21.15.152:0/4266735313 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4890045ad0 con 0x7f48900f5410 2023-12-06T03:58:54.681 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.679+0000 7f488d7fa700 1 -- 172.21.15.152:0/4266735313 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f488401db70 con 0x7f48900f5410 2023-12-06T03:58:54.681 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:54.681 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:54.683 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.681+0000 7f4896aa5700 1 -- 172.21.15.152:0/4266735313 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f488002ff60 msgr2=0x7f4880032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:54.683 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.681+0000 7f4896aa5700 1 --2- 172.21.15.152:0/4266735313 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f488002ff60 0x7f4880032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f487c0096a0 tx=0x7f487c006b40).stop 2023-12-06T03:58:54.683 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.681+0000 7f4896aa5700 1 -- 172.21.15.152:0/4266735313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48900f5410 msgr2=0x7f4890109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:54.683 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.681+0000 7f4896aa5700 1 --2- 172.21.15.152:0/4266735313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48900f5410 0x7f4890109450 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f4884013510 tx=0x7f488400b040).stop 2023-12-06T03:58:54.683 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.681+0000 7f4896aa5700 1 -- 172.21.15.152:0/4266735313 shutdown_connections 2023-12-06T03:58:54.683 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.681+0000 7f4896aa5700 1 --2- 172.21.15.152:0/4266735313 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f488002ff60 0x7f4880032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:54.683 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.681+0000 7f4896aa5700 1 --2- 172.21.15.152:0/4266735313 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f48900f5410 0x7f4890109450 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:54.683 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.681+0000 7f4896aa5700 1 -- 172.21.15.152:0/4266735313 >> 172.21.15.152:0/4266735313 conn(0x7f48900f0b70 msgr2=0x7f48900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:54.683 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.681+0000 7f4896aa5700 1 -- 172.21.15.152:0/4266735313 shutdown_connections 2023-12-06T03:58:54.683 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:54.681+0000 7f4896aa5700 1 -- 172.21.15.152:0/4266735313 wait complete. 2023-12-06T03:58:54.684 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:54.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:54.521812+0000 mon.smithi132 (mon.0) 257 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/mirror_snapshot_schedule"}]: dispatch 2023-12-06T03:58:54.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:54.530072+0000 mon.smithi132 (mon.0) 258 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/trash_purge_schedule"}]: dispatch 2023-12-06T03:58:55.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:54.440805+0000 mgr.smithi132.gokxja (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:55.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:54.680941+0000 mon.smithi132 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.152:0/4266735313' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:56.056 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:56.056 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:57.215 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.212+0000 7fadd73c5700 1 -- 172.21.15.152:0/594345103 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fadd00f5410 msgr2=0x7fadd00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:57.215 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.212+0000 7fadd73c5700 1 --2- 172.21.15.152:0/594345103 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fadd00f5410 0x7fadd00f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fadc4009fb0 tx=0x7fadc4009b30).stop 2023-12-06T03:58:57.215 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.212+0000 7fadd73c5700 1 -- 172.21.15.152:0/594345103 shutdown_connections 2023-12-06T03:58:57.215 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.212+0000 7fadd73c5700 1 --2- 172.21.15.152:0/594345103 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fadd00f5410 0x7fadd00f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:57.215 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.212+0000 7fadd73c5700 1 -- 172.21.15.152:0/594345103 >> 172.21.15.152:0/594345103 conn(0x7fadd00f0b70 msgr2=0x7fadd00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:57.215 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.212+0000 7fadd73c5700 1 -- 172.21.15.152:0/594345103 shutdown_connections 2023-12-06T03:58:57.216 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.212+0000 7fadd73c5700 1 -- 172.21.15.152:0/594345103 wait complete. 2023-12-06T03:58:57.216 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.213+0000 7fadd73c5700 1 Processor -- start 2023-12-06T03:58:57.216 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.213+0000 7fadd73c5700 1 -- start start 2023-12-06T03:58:57.216 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.213+0000 7fadd73c5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fadd00f5410 0x7fadd0105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:57.216 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.213+0000 7fadd73c5700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadd0105af0 con 0x7fadd00f5410 2023-12-06T03:58:57.216 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.213+0000 7fadd5161700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fadd00f5410 0x7fadd0105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:57.216 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.213+0000 7fadd5161700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fadd00f5410 0x7fadd0105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:40154/0 (socket says 172.21.15.152:40154) 2023-12-06T03:58:57.216 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.213+0000 7fadd5161700 1 -- 172.21.15.152:0/4287364046 learned_addr learned my addr 172.21.15.152:0/4287364046 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:57.216 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.213+0000 7fadd5161700 1 -- 172.21.15.152:0/4287364046 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadc4005040 con 0x7fadd00f5410 2023-12-06T03:58:57.216 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.214+0000 7fadd5161700 1 --2- 172.21.15.152:0/4287364046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fadd00f5410 0x7fadd0105600 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fadc4014000 tx=0x7fadc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:57.217 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.214+0000 7fadc1ffb700 1 -- 172.21.15.152:0/4287364046 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadc40095f0 con 0x7fadd00f5410 2023-12-06T03:58:57.217 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.214+0000 7fadd73c5700 1 -- 172.21.15.152:0/4287364046 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadd0105cf0 con 0x7fadd00f5410 2023-12-06T03:58:57.217 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.214+0000 7fadc1ffb700 1 -- 172.21.15.152:0/4287364046 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fadc4007e70 con 0x7fadd00f5410 2023-12-06T03:58:57.217 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.214+0000 7fadc1ffb700 1 -- 172.21.15.152:0/4287364046 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadc4014ea0 con 0x7fadd00f5410 2023-12-06T03:58:57.217 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.214+0000 7fadd73c5700 1 -- 172.21.15.152:0/4287364046 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadd0106180 con 0x7fadd00f5410 2023-12-06T03:58:57.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.215+0000 7fadc1ffb700 1 -- 172.21.15.152:0/4287364046 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fadc4025070 con 0x7fadd00f5410 2023-12-06T03:58:57.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.215+0000 7fadc1ffb700 1 --2- 172.21.15.152:0/4287364046 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fadbc02ff60 0x7fadbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:57.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.215+0000 7fadc1ffb700 1 -- 172.21.15.152:0/4287364046 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fadc4040a40 con 0x7fadd00f5410 2023-12-06T03:58:57.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.215+0000 7fadd73c5700 1 -- 172.21.15.152:0/4287364046 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadd0045ad0 con 0x7fadd00f5410 2023-12-06T03:58:57.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.215+0000 7fadd4960700 1 --2- 172.21.15.152:0/4287364046 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fadbc02ff60 0x7fadbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:57.218 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.216+0000 7fadd4960700 1 --2- 172.21.15.152:0/4287364046 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fadbc02ff60 0x7fadbc032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fadcc0096a0 tx=0x7fadcc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:57.221 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.220+0000 7fadc1ffb700 1 -- 172.21.15.152:0/4287364046 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fadc4026350 con 0x7fadd00f5410 2023-12-06T03:58:57.426 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.424+0000 7fadd73c5700 1 -- 172.21.15.152:0/4287364046 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fadd00faf50 con 0x7fadd00f5410 2023-12-06T03:58:57.427 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.425+0000 7fadc1ffb700 1 -- 172.21.15.152:0/4287364046 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fadc4012330 con 0x7fadd00f5410 2023-12-06T03:58:57.427 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:58:57.427 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:58:57.429 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.427+0000 7fadd73c5700 1 -- 172.21.15.152:0/4287364046 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fadbc02ff60 msgr2=0x7fadbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:57.429 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.427+0000 7fadd73c5700 1 --2- 172.21.15.152:0/4287364046 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fadbc02ff60 0x7fadbc032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fadcc0096a0 tx=0x7fadcc006b40).stop 2023-12-06T03:58:57.429 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.427+0000 7fadd73c5700 1 -- 172.21.15.152:0/4287364046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fadd00f5410 msgr2=0x7fadd0105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:57.430 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.427+0000 7fadd73c5700 1 --2- 172.21.15.152:0/4287364046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fadd00f5410 0x7fadd0105600 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fadc4014000 tx=0x7fadc400b040).stop 2023-12-06T03:58:57.430 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.427+0000 7fadd73c5700 1 -- 172.21.15.152:0/4287364046 shutdown_connections 2023-12-06T03:58:57.430 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.427+0000 7fadd73c5700 1 --2- 172.21.15.152:0/4287364046 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fadbc02ff60 0x7fadbc032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:57.433 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.427+0000 7fadd73c5700 1 --2- 172.21.15.152:0/4287364046 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fadd00f5410 0x7fadd0105600 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:57.433 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.427+0000 7fadd73c5700 1 -- 172.21.15.152:0/4287364046 >> 172.21.15.152:0/4287364046 conn(0x7fadd00f0b70 msgr2=0x7fadd00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:57.434 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.427+0000 7fadd73c5700 1 -- 172.21.15.152:0/4287364046 shutdown_connections 2023-12-06T03:58:57.434 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:57.427+0000 7fadd73c5700 1 -- 172.21.15.152:0/4287364046 wait complete. 2023-12-06T03:58:57.434 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:58:57.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:56.441084+0000 mgr.smithi132.gokxja (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:57.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:58:57.426675+0000 mon.smithi132 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.152:0/4287364046' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:58:58.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:58:58.802 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:58:59.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:58:59 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:58:58.441433+0000 mgr.smithi132.gokxja (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:58:59.920 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.919+0000 7ff2ee670700 1 -- 172.21.15.152:0/1342661182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2e80f5410 msgr2=0x7ff2e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:58:59.920 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.919+0000 7ff2ee670700 1 --2- 172.21.15.152:0/1342661182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2e80f5410 0x7ff2e80f5830 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7ff2d8009fb0 tx=0x7ff2d8009b30).stop 2023-12-06T03:58:59.921 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.919+0000 7ff2ee670700 1 -- 172.21.15.152:0/1342661182 shutdown_connections 2023-12-06T03:58:59.921 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.919+0000 7ff2ee670700 1 --2- 172.21.15.152:0/1342661182 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2e80f5410 0x7ff2e80f5830 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:58:59.921 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.919+0000 7ff2ee670700 1 -- 172.21.15.152:0/1342661182 >> 172.21.15.152:0/1342661182 conn(0x7ff2e80f0b70 msgr2=0x7ff2e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:58:59.921 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.919+0000 7ff2ee670700 1 -- 172.21.15.152:0/1342661182 shutdown_connections 2023-12-06T03:58:59.921 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.919+0000 7ff2ee670700 1 -- 172.21.15.152:0/1342661182 wait complete. 2023-12-06T03:58:59.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.920+0000 7ff2ee670700 1 Processor -- start 2023-12-06T03:58:59.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.920+0000 7ff2ee670700 1 -- start start 2023-12-06T03:58:59.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.921+0000 7ff2ee670700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2e80f5410 0x7ff2e8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:59.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.921+0000 7ff2ee670700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2e8060240 con 0x7ff2e80f5410 2023-12-06T03:58:59.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.921+0000 7ff2e7fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2e80f5410 0x7ff2e8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:59.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.921+0000 7ff2e7fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2e80f5410 0x7ff2e8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:55248/0 (socket says 172.21.15.152:55248) 2023-12-06T03:58:59.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.921+0000 7ff2e7fff700 1 -- 172.21.15.152:0/3200270801 learned_addr learned my addr 172.21.15.152:0/3200270801 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:58:59.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.921+0000 7ff2e7fff700 1 -- 172.21.15.152:0/3200270801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2d8005040 con 0x7ff2e80f5410 2023-12-06T03:58:59.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.921+0000 7ff2e7fff700 1 --2- 172.21.15.152:0/3200270801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2e80f5410 0x7ff2e8109430 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7ff2d8013510 tx=0x7ff2d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:59.924 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.921+0000 7ff2e4ff9700 1 -- 172.21.15.152:0/3200270801 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2d8009320 con 0x7ff2e80f5410 2023-12-06T03:58:59.924 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.921+0000 7ff2ee670700 1 -- 172.21.15.152:0/3200270801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2e8060440 con 0x7ff2e80f5410 2023-12-06T03:58:59.924 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.921+0000 7ff2e4ff9700 1 -- 172.21.15.152:0/3200270801 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2d8013f00 con 0x7ff2e80f5410 2023-12-06T03:58:59.924 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.921+0000 7ff2e4ff9700 1 -- 172.21.15.152:0/3200270801 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2d8014d90 con 0x7ff2e80f5410 2023-12-06T03:58:59.924 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.921+0000 7ff2ee670700 1 -- 172.21.15.152:0/3200270801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2e8109ae0 con 0x7ff2e80f5410 2023-12-06T03:58:59.924 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.922+0000 7ff2e4ff9700 1 -- 172.21.15.152:0/3200270801 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2d80095f0 con 0x7ff2e80f5410 2023-12-06T03:58:59.924 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.922+0000 7ff2ee670700 1 -- 172.21.15.152:0/3200270801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2e8045ad0 con 0x7ff2e80f5410 2023-12-06T03:58:59.924 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.922+0000 7ff2e4ff9700 1 --2- 172.21.15.152:0/3200270801 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff2d402ff60 0x7ff2d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:58:59.924 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.922+0000 7ff2e4ff9700 1 -- 172.21.15.152:0/3200270801 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2d8041080 con 0x7ff2e80f5410 2023-12-06T03:58:59.924 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.923+0000 7ff2e77fe700 1 --2- 172.21.15.152:0/3200270801 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff2d402ff60 0x7ff2d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:58:59.925 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.923+0000 7ff2e77fe700 1 --2- 172.21.15.152:0/3200270801 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff2d402ff60 0x7ff2d4032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7ff2d00096a0 tx=0x7ff2d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:58:59.927 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:58:59.927+0000 7ff2e4ff9700 1 -- 172.21.15.152:0/3200270801 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2d8014420 con 0x7ff2e80f5410 2023-12-06T03:59:00.133 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:00.133+0000 7ff2ee670700 1 -- 172.21.15.152:0/3200270801 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2e80fae60 con 0x7ff2e80f5410 2023-12-06T03:59:00.134 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:00.133+0000 7ff2e4ff9700 1 -- 172.21.15.152:0/3200270801 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff2d8010230 con 0x7ff2e80f5410 2023-12-06T03:59:00.134 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:00.134 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:00.135 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:00.135+0000 7ff2ee670700 1 -- 172.21.15.152:0/3200270801 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff2d402ff60 msgr2=0x7ff2d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:00.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:00.135+0000 7ff2ee670700 1 --2- 172.21.15.152:0/3200270801 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff2d402ff60 0x7ff2d4032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7ff2d00096a0 tx=0x7ff2d0006b40).stop 2023-12-06T03:59:00.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:00.135+0000 7ff2ee670700 1 -- 172.21.15.152:0/3200270801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2e80f5410 msgr2=0x7ff2e8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:00.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:00.135+0000 7ff2ee670700 1 --2- 172.21.15.152:0/3200270801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2e80f5410 0x7ff2e8109430 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7ff2d8013510 tx=0x7ff2d800b040).stop 2023-12-06T03:59:00.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:00.135+0000 7ff2ee670700 1 -- 172.21.15.152:0/3200270801 shutdown_connections 2023-12-06T03:59:00.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:00.135+0000 7ff2ee670700 1 --2- 172.21.15.152:0/3200270801 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff2d402ff60 0x7ff2d4032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:00.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:00.135+0000 7ff2ee670700 1 --2- 172.21.15.152:0/3200270801 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff2e80f5410 0x7ff2e8109430 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:00.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:00.135+0000 7ff2ee670700 1 -- 172.21.15.152:0/3200270801 >> 172.21.15.152:0/3200270801 conn(0x7ff2e80f0b70 msgr2=0x7ff2e80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:00.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:00.135+0000 7ff2ee670700 1 -- 172.21.15.152:0/3200270801 shutdown_connections 2023-12-06T03:59:00.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:00.135+0000 7ff2ee670700 1 -- 172.21.15.152:0/3200270801 wait complete. 2023-12-06T03:59:00.136 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:00.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:00 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:00.133754+0000 mon.smithi132 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.152:0/3200270801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:01.522 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:01.522 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:01.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:01 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:00.441767+0000 mgr.smithi132.gokxja (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:02.659 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.658+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/2923905412 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0a80f6510 msgr2=0x7fb0a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:02.659 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.658+0000 7fb0ae3c4700 1 --2- 172.21.15.152:0/2923905412 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0a80f6510 0x7fb0a80f6930 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fb098009fb0 tx=0x7fb098009b30).stop 2023-12-06T03:59:02.659 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.658+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/2923905412 shutdown_connections 2023-12-06T03:59:02.660 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.658+0000 7fb0ae3c4700 1 --2- 172.21.15.152:0/2923905412 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0a80f6510 0x7fb0a80f6930 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:02.660 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.658+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/2923905412 >> 172.21.15.152:0/2923905412 conn(0x7fb0a80f1c70 msgr2=0x7fb0a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:02.660 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.658+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/2923905412 shutdown_connections 2023-12-06T03:59:02.660 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.658+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/2923905412 wait complete. 2023-12-06T03:59:02.660 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.659+0000 7fb0ae3c4700 1 Processor -- start 2023-12-06T03:59:02.661 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.659+0000 7fb0ae3c4700 1 -- start start 2023-12-06T03:59:02.661 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.659+0000 7fb0ae3c4700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0a80f6510 0x7fb0a8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:02.661 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.659+0000 7fb0ae3c4700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a8106c20 con 0x7fb0a80f6510 2023-12-06T03:59:02.661 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.660+0000 7fb0a7fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0a80f6510 0x7fb0a8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:02.661 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.660+0000 7fb0a7fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0a80f6510 0x7fb0a8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:55270/0 (socket says 172.21.15.152:55270) 2023-12-06T03:59:02.661 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.660+0000 7fb0a7fff700 1 -- 172.21.15.152:0/3896011865 learned_addr learned my addr 172.21.15.152:0/3896011865 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:02.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.660+0000 7fb0a7fff700 1 -- 172.21.15.152:0/3896011865 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb098005040 con 0x7fb0a80f6510 2023-12-06T03:59:02.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.660+0000 7fb0a7fff700 1 --2- 172.21.15.152:0/3896011865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0a80f6510 0x7fb0a8106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fb098014000 tx=0x7fb09800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:02.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.660+0000 7fb0a4ff9700 1 -- 172.21.15.152:0/3896011865 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb0980095f0 con 0x7fb0a80f6510 2023-12-06T03:59:02.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.660+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/3896011865 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a8106e20 con 0x7fb0a80f6510 2023-12-06T03:59:02.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.660+0000 7fb0a4ff9700 1 -- 172.21.15.152:0/3896011865 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb098007e70 con 0x7fb0a80f6510 2023-12-06T03:59:02.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.660+0000 7fb0a4ff9700 1 -- 172.21.15.152:0/3896011865 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb098014ea0 con 0x7fb0a80f6510 2023-12-06T03:59:02.662 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.660+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/3896011865 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a81072b0 con 0x7fb0a80f6510 2023-12-06T03:59:02.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.661+0000 7fb0a4ff9700 1 -- 172.21.15.152:0/3896011865 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb098025070 con 0x7fb0a80f6510 2023-12-06T03:59:02.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.661+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/3896011865 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a8045ad0 con 0x7fb0a80f6510 2023-12-06T03:59:02.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.661+0000 7fb0a4ff9700 1 --2- 172.21.15.152:0/3896011865 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb09002ff60 0x7fb090032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:02.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.661+0000 7fb0a4ff9700 1 -- 172.21.15.152:0/3896011865 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb098040a30 con 0x7fb0a80f6510 2023-12-06T03:59:02.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.661+0000 7fb0a77fe700 1 --2- 172.21.15.152:0/3896011865 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb09002ff60 0x7fb090032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:02.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.662+0000 7fb0a77fe700 1 --2- 172.21.15.152:0/3896011865 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb09002ff60 0x7fb090032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fb09c0096a0 tx=0x7fb09c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:02.667 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.666+0000 7fb0a4ff9700 1 -- 172.21.15.152:0/3896011865 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0980079c0 con 0x7fb0a80f6510 2023-12-06T03:59:02.876 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.875+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/3896011865 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0a80fbf60 con 0x7fb0a80f6510 2023-12-06T03:59:02.876 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.876+0000 7fb0a4ff9700 1 -- 172.21.15.152:0/3896011865 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb09800f300 con 0x7fb0a80f6510 2023-12-06T03:59:02.876 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:02.877 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:02.878 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.877+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/3896011865 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb09002ff60 msgr2=0x7fb090032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:02.878 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.877+0000 7fb0ae3c4700 1 --2- 172.21.15.152:0/3896011865 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb09002ff60 0x7fb090032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fb09c0096a0 tx=0x7fb09c006b40).stop 2023-12-06T03:59:02.878 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.877+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/3896011865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0a80f6510 msgr2=0x7fb0a8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:02.878 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.877+0000 7fb0ae3c4700 1 --2- 172.21.15.152:0/3896011865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0a80f6510 0x7fb0a8106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fb098014000 tx=0x7fb09800b040).stop 2023-12-06T03:59:02.878 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.877+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/3896011865 shutdown_connections 2023-12-06T03:59:02.878 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.877+0000 7fb0ae3c4700 1 --2- 172.21.15.152:0/3896011865 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb09002ff60 0x7fb090032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:02.879 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.877+0000 7fb0ae3c4700 1 --2- 172.21.15.152:0/3896011865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb0a80f6510 0x7fb0a8106730 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:02.879 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.877+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/3896011865 >> 172.21.15.152:0/3896011865 conn(0x7fb0a80f1c70 msgr2=0x7fb0a80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:02.879 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.877+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/3896011865 shutdown_connections 2023-12-06T03:59:02.879 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:02.877+0000 7fb0ae3c4700 1 -- 172.21.15.152:0/3896011865 wait complete. 2023-12-06T03:59:02.879 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:03.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:02.442171+0000 mgr.smithi132.gokxja (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:03.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:02.876051+0000 mon.smithi132 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.152:0/3896011865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:04.239 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:04.239 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:05.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.340+0000 7ffbc714d700 1 -- 172.21.15.152:0/3030646514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc00f51a0 msgr2=0x7ffbc00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:05.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.340+0000 7ffbc714d700 1 --2- 172.21.15.152:0/3030646514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc00f51a0 0x7ffbc00f55c0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7ffbb0009fb0 tx=0x7ffbb0009b30).stop 2023-12-06T03:59:05.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.340+0000 7ffbc714d700 1 -- 172.21.15.152:0/3030646514 shutdown_connections 2023-12-06T03:59:05.341 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.340+0000 7ffbc714d700 1 --2- 172.21.15.152:0/3030646514 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc00f51a0 0x7ffbc00f55c0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:05.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.340+0000 7ffbc714d700 1 -- 172.21.15.152:0/3030646514 >> 172.21.15.152:0/3030646514 conn(0x7ffbc00f09a0 msgr2=0x7ffbc00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:05.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.340+0000 7ffbc714d700 1 -- 172.21.15.152:0/3030646514 shutdown_connections 2023-12-06T03:59:05.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.340+0000 7ffbc714d700 1 -- 172.21.15.152:0/3030646514 wait complete. 2023-12-06T03:59:05.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.340+0000 7ffbc714d700 1 Processor -- start 2023-12-06T03:59:05.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.341+0000 7ffbc714d700 1 -- start start 2023-12-06T03:59:05.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.341+0000 7ffbc714d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc00f51a0 0x7ffbc00fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:05.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.341+0000 7ffbc714d700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbc00fb5a0 con 0x7ffbc00f51a0 2023-12-06T03:59:05.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.341+0000 7ffbc614b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc00f51a0 0x7ffbc00fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:05.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.341+0000 7ffbc614b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc00f51a0 0x7ffbc00fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:55282/0 (socket says 172.21.15.152:55282) 2023-12-06T03:59:05.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.341+0000 7ffbc614b700 1 -- 172.21.15.152:0/293329833 learned_addr learned my addr 172.21.15.152:0/293329833 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:05.344 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.341+0000 7ffbc614b700 1 -- 172.21.15.152:0/293329833 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb0005040 con 0x7ffbc00f51a0 2023-12-06T03:59:05.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.341+0000 7ffbc614b700 1 --2- 172.21.15.152:0/293329833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc00f51a0 0x7ffbc00fd120 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7ffbb0009830 tx=0x7ffbb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:05.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.341+0000 7ffbb6ffd700 1 -- 172.21.15.152:0/293329833 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbb0009320 con 0x7ffbc00f51a0 2023-12-06T03:59:05.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.341+0000 7ffbc714d700 1 -- 172.21.15.152:0/293329833 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbc00fb740 con 0x7ffbc00f51a0 2023-12-06T03:59:05.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.341+0000 7ffbb6ffd700 1 -- 172.21.15.152:0/293329833 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffbb00136c0 con 0x7ffbc00f51a0 2023-12-06T03:59:05.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.341+0000 7ffbb6ffd700 1 -- 172.21.15.152:0/293329833 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbb001ee90 con 0x7ffbc00f51a0 2023-12-06T03:59:05.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.341+0000 7ffbc714d700 1 -- 172.21.15.152:0/293329833 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbc00fbbd0 con 0x7ffbc00f51a0 2023-12-06T03:59:05.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.342+0000 7ffbb6ffd700 1 -- 172.21.15.152:0/293329833 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffbb001e420 con 0x7ffbc00f51a0 2023-12-06T03:59:05.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.342+0000 7ffbc714d700 1 -- 172.21.15.152:0/293329833 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbc0059100 con 0x7ffbc00f51a0 2023-12-06T03:59:05.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.342+0000 7ffbb6ffd700 1 --2- 172.21.15.152:0/293329833 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffbac02ff10 0x7ffbac0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:05.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.342+0000 7ffbb6ffd700 1 -- 172.21.15.152:0/293329833 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffbb0041120 con 0x7ffbc00f51a0 2023-12-06T03:59:05.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.343+0000 7ffbc594a700 1 --2- 172.21.15.152:0/293329833 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffbac02ff10 0x7ffbac0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:05.345 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.343+0000 7ffbc594a700 1 --2- 172.21.15.152:0/293329833 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffbac02ff10 0x7ffbac0323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7ffbbc0096a0 tx=0x7ffbbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:05.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.347+0000 7ffbb6ffd700 1 -- 172.21.15.152:0/293329833 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffbb00095f0 con 0x7ffbc00f51a0 2023-12-06T03:59:05.551 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.551+0000 7ffbc714d700 1 -- 172.21.15.152:0/293329833 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffbc00fabe0 con 0x7ffbc00f51a0 2023-12-06T03:59:05.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.551+0000 7ffbb6ffd700 1 -- 172.21.15.152:0/293329833 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffbb001cb10 con 0x7ffbc00f51a0 2023-12-06T03:59:05.552 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:05.552 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:05.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.553+0000 7ffbc714d700 1 -- 172.21.15.152:0/293329833 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffbac02ff10 msgr2=0x7ffbac0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:05.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.553+0000 7ffbc714d700 1 --2- 172.21.15.152:0/293329833 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffbac02ff10 0x7ffbac0323d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7ffbbc0096a0 tx=0x7ffbbc006b40).stop 2023-12-06T03:59:05.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.553+0000 7ffbc714d700 1 -- 172.21.15.152:0/293329833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc00f51a0 msgr2=0x7ffbc00fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:05.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.553+0000 7ffbc714d700 1 --2- 172.21.15.152:0/293329833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc00f51a0 0x7ffbc00fd120 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7ffbb0009830 tx=0x7ffbb000b040).stop 2023-12-06T03:59:05.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.553+0000 7ffbc714d700 1 -- 172.21.15.152:0/293329833 shutdown_connections 2023-12-06T03:59:05.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.553+0000 7ffbc714d700 1 --2- 172.21.15.152:0/293329833 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffbac02ff10 0x7ffbac0323d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:05.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.553+0000 7ffbc714d700 1 --2- 172.21.15.152:0/293329833 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbc00f51a0 0x7ffbc00fd120 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:05.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.553+0000 7ffbc714d700 1 -- 172.21.15.152:0/293329833 >> 172.21.15.152:0/293329833 conn(0x7ffbc00f09a0 msgr2=0x7ffbc005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:05.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.553+0000 7ffbc714d700 1 -- 172.21.15.152:0/293329833 shutdown_connections 2023-12-06T03:59:05.554 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:05.553+0000 7ffbc714d700 1 -- 172.21.15.152:0/293329833 wait complete. 2023-12-06T03:59:05.555 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:05.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:04.442480+0000 mgr.smithi132.gokxja (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:05.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:05.551888+0000 mon.smithi132 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.152:0/293329833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:06.960 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:06.960 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:07.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:07 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:06.442810+0000 mgr.smithi132.gokxja (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:08.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.102+0000 7fdf258fd700 1 -- 172.21.15.152:0/1578864601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf200f51a0 msgr2=0x7fdf200f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:08.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.102+0000 7fdf258fd700 1 --2- 172.21.15.152:0/1578864601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf200f51a0 0x7fdf200f55c0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7fdf14009fb0 tx=0x7fdf14009b30).stop 2023-12-06T03:59:08.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.103+0000 7fdf258fd700 1 -- 172.21.15.152:0/1578864601 shutdown_connections 2023-12-06T03:59:08.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.103+0000 7fdf258fd700 1 --2- 172.21.15.152:0/1578864601 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf200f51a0 0x7fdf200f55c0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:08.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.103+0000 7fdf258fd700 1 -- 172.21.15.152:0/1578864601 >> 172.21.15.152:0/1578864601 conn(0x7fdf200f09a0 msgr2=0x7fdf200f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:08.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.103+0000 7fdf258fd700 1 -- 172.21.15.152:0/1578864601 shutdown_connections 2023-12-06T03:59:08.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.103+0000 7fdf258fd700 1 -- 172.21.15.152:0/1578864601 wait complete. 2023-12-06T03:59:08.104 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.103+0000 7fdf258fd700 1 Processor -- start 2023-12-06T03:59:08.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.103+0000 7fdf258fd700 1 -- start start 2023-12-06T03:59:08.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.103+0000 7fdf258fd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf200f51a0 0x7fdf200fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:08.105 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.103+0000 7fdf258fd700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf200fb5a0 con 0x7fdf200f51a0 2023-12-06T03:59:08.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.104+0000 7fdf248fb700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf200f51a0 0x7fdf200fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:08.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.104+0000 7fdf248fb700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf200f51a0 0x7fdf200fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:55298/0 (socket says 172.21.15.152:55298) 2023-12-06T03:59:08.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.104+0000 7fdf248fb700 1 -- 172.21.15.152:0/4214357361 learned_addr learned my addr 172.21.15.152:0/4214357361 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:08.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.104+0000 7fdf248fb700 1 -- 172.21.15.152:0/4214357361 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf14005040 con 0x7fdf200f51a0 2023-12-06T03:59:08.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.104+0000 7fdf248fb700 1 --2- 172.21.15.152:0/4214357361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf200f51a0 0x7fdf200fd120 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fdf14013510 tx=0x7fdf1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:08.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.104+0000 7fdf1d7fa700 1 -- 172.21.15.152:0/4214357361 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf14009320 con 0x7fdf200f51a0 2023-12-06T03:59:08.106 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.104+0000 7fdf1d7fa700 1 -- 172.21.15.152:0/4214357361 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf14013f00 con 0x7fdf200f51a0 2023-12-06T03:59:08.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.104+0000 7fdf258fd700 1 -- 172.21.15.152:0/4214357361 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf200fb740 con 0x7fdf200f51a0 2023-12-06T03:59:08.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.104+0000 7fdf1d7fa700 1 -- 172.21.15.152:0/4214357361 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf14014dc0 con 0x7fdf200f51a0 2023-12-06T03:59:08.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.104+0000 7fdf258fd700 1 -- 172.21.15.152:0/4214357361 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf200fbbd0 con 0x7fdf200f51a0 2023-12-06T03:59:08.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.105+0000 7fdf1d7fa700 1 -- 172.21.15.152:0/4214357361 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf140095f0 con 0x7fdf200f51a0 2023-12-06T03:59:08.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.105+0000 7fdf258fd700 1 -- 172.21.15.152:0/4214357361 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf20059100 con 0x7fdf200f51a0 2023-12-06T03:59:08.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.105+0000 7fdf1d7fa700 1 --2- 172.21.15.152:0/4214357361 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf1002ff60 0x7fdf10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:08.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.105+0000 7fdf1d7fa700 1 -- 172.21.15.152:0/4214357361 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf140410e0 con 0x7fdf200f51a0 2023-12-06T03:59:08.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.106+0000 7fdf1ffff700 1 --2- 172.21.15.152:0/4214357361 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf1002ff60 0x7fdf10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:08.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.106+0000 7fdf1ffff700 1 --2- 172.21.15.152:0/4214357361 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf1002ff60 0x7fdf10032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fdf0c0096a0 tx=0x7fdf0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:08.110 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.110+0000 7fdf1d7fa700 1 -- 172.21.15.152:0/4214357361 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf14014420 con 0x7fdf200f51a0 2023-12-06T03:59:08.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.315+0000 7fdf258fd700 1 -- 172.21.15.152:0/4214357361 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf20045ad0 con 0x7fdf200f51a0 2023-12-06T03:59:08.316 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.315+0000 7fdf1d7fa700 1 -- 172.21.15.152:0/4214357361 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdf1401c920 con 0x7fdf200f51a0 2023-12-06T03:59:08.316 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:08.316 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:08.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.317+0000 7fdf258fd700 1 -- 172.21.15.152:0/4214357361 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf1002ff60 msgr2=0x7fdf10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:08.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.317+0000 7fdf258fd700 1 --2- 172.21.15.152:0/4214357361 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf1002ff60 0x7fdf10032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fdf0c0096a0 tx=0x7fdf0c006b40).stop 2023-12-06T03:59:08.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.317+0000 7fdf258fd700 1 -- 172.21.15.152:0/4214357361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf200f51a0 msgr2=0x7fdf200fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:08.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.317+0000 7fdf258fd700 1 --2- 172.21.15.152:0/4214357361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf200f51a0 0x7fdf200fd120 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7fdf14013510 tx=0x7fdf1400b040).stop 2023-12-06T03:59:08.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.317+0000 7fdf258fd700 1 -- 172.21.15.152:0/4214357361 shutdown_connections 2023-12-06T03:59:08.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.317+0000 7fdf258fd700 1 --2- 172.21.15.152:0/4214357361 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fdf1002ff60 0x7fdf10032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:08.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.317+0000 7fdf258fd700 1 --2- 172.21.15.152:0/4214357361 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fdf200f51a0 0x7fdf200fd120 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:08.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.317+0000 7fdf258fd700 1 -- 172.21.15.152:0/4214357361 >> 172.21.15.152:0/4214357361 conn(0x7fdf200f09a0 msgr2=0x7fdf2005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:08.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.317+0000 7fdf258fd700 1 -- 172.21.15.152:0/4214357361 shutdown_connections 2023-12-06T03:59:08.318 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:08.317+0000 7fdf258fd700 1 -- 172.21.15.152:0/4214357361 wait complete. 2023-12-06T03:59:08.319 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:08.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:08 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:08.315991+0000 mon.smithi132 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.152:0/4214357361' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:09.702 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:09.702 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:09.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:09 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:08.443138+0000 mgr.smithi132.gokxja (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:10.755 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.753+0000 7f057b0c3700 1 -- 172.21.15.152:0/2407075422 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05740f62a0 msgr2=0x7f05740f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:10.755 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.753+0000 7f057b0c3700 1 --2- 172.21.15.152:0/2407075422 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05740f62a0 0x7f05740f66c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f0570009fb0 tx=0x7f0570009b30).stop 2023-12-06T03:59:10.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.753+0000 7f057b0c3700 1 -- 172.21.15.152:0/2407075422 shutdown_connections 2023-12-06T03:59:10.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.753+0000 7f057b0c3700 1 --2- 172.21.15.152:0/2407075422 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05740f62a0 0x7f05740f66c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:10.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.753+0000 7f057b0c3700 1 -- 172.21.15.152:0/2407075422 >> 172.21.15.152:0/2407075422 conn(0x7f05740f1aa0 msgr2=0x7f05740f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:10.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.753+0000 7f057b0c3700 1 -- 172.21.15.152:0/2407075422 shutdown_connections 2023-12-06T03:59:10.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.753+0000 7f057b0c3700 1 -- 172.21.15.152:0/2407075422 wait complete. 2023-12-06T03:59:10.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.753+0000 7f057b0c3700 1 Processor -- start 2023-12-06T03:59:10.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.753+0000 7f057b0c3700 1 -- start start 2023-12-06T03:59:10.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.753+0000 7f057b0c3700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05740f62a0 0x7f05741064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:10.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.753+0000 7f057b0c3700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05741069a0 con 0x7f05740f62a0 2023-12-06T03:59:10.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.754+0000 7f057a0c1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05740f62a0 0x7f05741064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:10.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.754+0000 7f057a0c1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05740f62a0 0x7f05741064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:34154/0 (socket says 172.21.15.152:34154) 2023-12-06T03:59:10.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.754+0000 7f057a0c1700 1 -- 172.21.15.152:0/450071902 learned_addr learned my addr 172.21.15.152:0/450071902 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:10.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.754+0000 7f057a0c1700 1 -- 172.21.15.152:0/450071902 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0570005040 con 0x7f05740f62a0 2023-12-06T03:59:10.756 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.754+0000 7f057a0c1700 1 --2- 172.21.15.152:0/450071902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05740f62a0 0x7f05741064b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f0570013510 tx=0x7f057000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:10.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.754+0000 7f056affd700 1 -- 172.21.15.152:0/450071902 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0570009320 con 0x7f05740f62a0 2023-12-06T03:59:10.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.754+0000 7f056affd700 1 -- 172.21.15.152:0/450071902 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0570013f00 con 0x7f05740f62a0 2023-12-06T03:59:10.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.754+0000 7f057b0c3700 1 -- 172.21.15.152:0/450071902 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0574106ba0 con 0x7f05740f62a0 2023-12-06T03:59:10.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.754+0000 7f056affd700 1 -- 172.21.15.152:0/450071902 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0570014dc0 con 0x7f05740f62a0 2023-12-06T03:59:10.757 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.754+0000 7f057b0c3700 1 -- 172.21.15.152:0/450071902 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0574107030 con 0x7f05740f62a0 2023-12-06T03:59:10.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.755+0000 7f056affd700 1 -- 172.21.15.152:0/450071902 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f057001d3d0 con 0x7f05740f62a0 2023-12-06T03:59:10.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.755+0000 7f056affd700 1 --2- 172.21.15.152:0/450071902 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f056002ff60 0x7f0560032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:10.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.755+0000 7f056affd700 1 -- 172.21.15.152:0/450071902 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0570041400 con 0x7f05740f62a0 2023-12-06T03:59:10.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.755+0000 7f057b0c3700 1 -- 172.21.15.152:0/450071902 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0574045ad0 con 0x7f05740f62a0 2023-12-06T03:59:10.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.755+0000 7f05798c0700 1 --2- 172.21.15.152:0/450071902 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f056002ff60 0x7f0560032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:10.758 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.756+0000 7f05798c0700 1 --2- 172.21.15.152:0/450071902 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f056002ff60 0x7f0560032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f05640096a0 tx=0x7f0564006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:10.760 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.760+0000 7f056affd700 1 -- 172.21.15.152:0/450071902 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0570012900 con 0x7f05740f62a0 2023-12-06T03:59:10.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.967+0000 7f057b0c3700 1 -- 172.21.15.152:0/450071902 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0574107d50 con 0x7f05740f62a0 2023-12-06T03:59:10.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.968+0000 7f056affd700 1 -- 172.21.15.152:0/450071902 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f057002b030 con 0x7f05740f62a0 2023-12-06T03:59:10.968 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:10.969 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:10.970 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.969+0000 7f057b0c3700 1 -- 172.21.15.152:0/450071902 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f056002ff60 msgr2=0x7f0560032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:10.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.969+0000 7f057b0c3700 1 --2- 172.21.15.152:0/450071902 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f056002ff60 0x7f0560032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f05640096a0 tx=0x7f0564006b40).stop 2023-12-06T03:59:10.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.969+0000 7f057b0c3700 1 -- 172.21.15.152:0/450071902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05740f62a0 msgr2=0x7f05741064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:10.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.969+0000 7f057b0c3700 1 --2- 172.21.15.152:0/450071902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05740f62a0 0x7f05741064b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f0570013510 tx=0x7f057000b040).stop 2023-12-06T03:59:10.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.969+0000 7f057b0c3700 1 -- 172.21.15.152:0/450071902 shutdown_connections 2023-12-06T03:59:10.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.969+0000 7f057b0c3700 1 --2- 172.21.15.152:0/450071902 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f056002ff60 0x7f0560032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:10.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.969+0000 7f057b0c3700 1 --2- 172.21.15.152:0/450071902 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f05740f62a0 0x7f05741064b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:10.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.969+0000 7f057b0c3700 1 -- 172.21.15.152:0/450071902 >> 172.21.15.152:0/450071902 conn(0x7f05740f1aa0 msgr2=0x7f05740f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:10.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.969+0000 7f057b0c3700 1 -- 172.21.15.152:0/450071902 shutdown_connections 2023-12-06T03:59:10.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:10.969+0000 7f057b0c3700 1 -- 172.21.15.152:0/450071902 wait complete. 2023-12-06T03:59:10.971 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:11.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:11 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:10.443494+0000 mgr.smithi132.gokxja (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:11.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:11 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:10.968127+0000 mon.smithi132 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.152:0/450071902' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:12.334 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:12.334 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:13.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.447+0000 7f7e0ffff700 1 -- 172.21.15.152:0/649009065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e100f51a0 msgr2=0x7f7e100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:13.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.448+0000 7f7e0ffff700 1 --2- 172.21.15.152:0/649009065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e100f51a0 0x7f7e100f55c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f7e00009fb0 tx=0x7f7e00009b30).stop 2023-12-06T03:59:13.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.448+0000 7f7e0ffff700 1 -- 172.21.15.152:0/649009065 shutdown_connections 2023-12-06T03:59:13.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.448+0000 7f7e0ffff700 1 --2- 172.21.15.152:0/649009065 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e100f51a0 0x7f7e100f55c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:13.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.448+0000 7f7e0ffff700 1 -- 172.21.15.152:0/649009065 >> 172.21.15.152:0/649009065 conn(0x7f7e100f09a0 msgr2=0x7f7e100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:13.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.448+0000 7f7e0ffff700 1 -- 172.21.15.152:0/649009065 shutdown_connections 2023-12-06T03:59:13.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.448+0000 7f7e0ffff700 1 -- 172.21.15.152:0/649009065 wait complete. 2023-12-06T03:59:13.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.448+0000 7f7e0ffff700 1 Processor -- start 2023-12-06T03:59:13.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.448+0000 7f7e0ffff700 1 -- start start 2023-12-06T03:59:13.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.448+0000 7f7e0ffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e100f51a0 0x7f7e101091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:13.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.448+0000 7f7e0ffff700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e10060220 con 0x7f7e100f51a0 2023-12-06T03:59:13.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.449+0000 7f7e0effd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e100f51a0 0x7f7e101091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:13.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.449+0000 7f7e0effd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e100f51a0 0x7f7e101091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:34174/0 (socket says 172.21.15.152:34174) 2023-12-06T03:59:13.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.449+0000 7f7e0effd700 1 -- 172.21.15.152:0/1812375432 learned_addr learned my addr 172.21.15.152:0/1812375432 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:13.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.449+0000 7f7e0effd700 1 -- 172.21.15.152:0/1812375432 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e00005040 con 0x7f7e100f51a0 2023-12-06T03:59:13.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.449+0000 7f7e0effd700 1 --2- 172.21.15.152:0/1812375432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e100f51a0 0x7f7e101091d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f7e00013510 tx=0x7f7e0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:13.451 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.449+0000 7f7df77fe700 1 -- 172.21.15.152:0/1812375432 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e00009320 con 0x7f7e100f51a0 2023-12-06T03:59:13.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.449+0000 7f7e0ffff700 1 -- 172.21.15.152:0/1812375432 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e10060420 con 0x7f7e100f51a0 2023-12-06T03:59:13.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.449+0000 7f7df77fe700 1 -- 172.21.15.152:0/1812375432 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7e00013f00 con 0x7f7e100f51a0 2023-12-06T03:59:13.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.449+0000 7f7df77fe700 1 -- 172.21.15.152:0/1812375432 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7e00014dc0 con 0x7f7e100f51a0 2023-12-06T03:59:13.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.449+0000 7f7e0ffff700 1 -- 172.21.15.152:0/1812375432 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e10109880 con 0x7f7e100f51a0 2023-12-06T03:59:13.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.450+0000 7f7df77fe700 1 -- 172.21.15.152:0/1812375432 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7e0001d3d0 con 0x7f7e100f51a0 2023-12-06T03:59:13.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.450+0000 7f7df77fe700 1 --2- 172.21.15.152:0/1812375432 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7df802ff60 0x7f7df8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:13.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.450+0000 7f7df77fe700 1 -- 172.21.15.152:0/1812375432 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7e00041120 con 0x7f7e100f51a0 2023-12-06T03:59:13.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.450+0000 7f7e0ffff700 1 -- 172.21.15.152:0/1812375432 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7e10059100 con 0x7f7e100f51a0 2023-12-06T03:59:13.452 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.450+0000 7f7e0e7fc700 1 --2- 172.21.15.152:0/1812375432 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7df802ff60 0x7f7df8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:13.453 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.451+0000 7f7e0e7fc700 1 --2- 172.21.15.152:0/1812375432 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7df802ff60 0x7f7df8032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f7e040096a0 tx=0x7f7e04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:13.456 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.455+0000 7f7df77fe700 1 -- 172.21.15.152:0/1812375432 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7e00012920 con 0x7f7e100f51a0 2023-12-06T03:59:13.663 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.662+0000 7f7e0ffff700 1 -- 172.21.15.152:0/1812375432 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7e10109f70 con 0x7f7e100f51a0 2023-12-06T03:59:13.664 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.663+0000 7f7df77fe700 1 -- 172.21.15.152:0/1812375432 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7e0002b020 con 0x7f7e100f51a0 2023-12-06T03:59:13.664 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:13.664 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:13.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.665+0000 7f7e0ffff700 1 -- 172.21.15.152:0/1812375432 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7df802ff60 msgr2=0x7f7df8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:13.665 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.665+0000 7f7e0ffff700 1 --2- 172.21.15.152:0/1812375432 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7df802ff60 0x7f7df8032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f7e040096a0 tx=0x7f7e04006b40).stop 2023-12-06T03:59:13.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.665+0000 7f7e0ffff700 1 -- 172.21.15.152:0/1812375432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e100f51a0 msgr2=0x7f7e101091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:13.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.665+0000 7f7e0ffff700 1 --2- 172.21.15.152:0/1812375432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e100f51a0 0x7f7e101091d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f7e00013510 tx=0x7f7e0000b040).stop 2023-12-06T03:59:13.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.665+0000 7f7e0ffff700 1 -- 172.21.15.152:0/1812375432 shutdown_connections 2023-12-06T03:59:13.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.665+0000 7f7e0ffff700 1 --2- 172.21.15.152:0/1812375432 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7df802ff60 0x7f7df8032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:13.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.665+0000 7f7e0ffff700 1 --2- 172.21.15.152:0/1812375432 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7e100f51a0 0x7f7e101091d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:13.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.665+0000 7f7e0ffff700 1 -- 172.21.15.152:0/1812375432 >> 172.21.15.152:0/1812375432 conn(0x7f7e100f09a0 msgr2=0x7f7e100f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:13.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.665+0000 7f7e0ffff700 1 -- 172.21.15.152:0/1812375432 shutdown_connections 2023-12-06T03:59:13.666 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:13.665+0000 7f7e0ffff700 1 -- 172.21.15.152:0/1812375432 wait complete. 2023-12-06T03:59:13.666 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:13.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:13 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:12.443902+0000 mgr.smithi132.gokxja (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:14.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:14 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:13.663450+0000 mon.smithi132 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.152:0/1812375432' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:15.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:15.039 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:15.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:15 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:14.444326+0000 mgr.smithi132.gokxja (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:16.156 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.155+0000 7feb6ab65700 1 -- 172.21.15.152:0/3748116389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb640f5410 msgr2=0x7feb640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:16.156 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.155+0000 7feb6ab65700 1 --2- 172.21.15.152:0/3748116389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb640f5410 0x7feb640f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7feb58009fb0 tx=0x7feb58009b30).stop 2023-12-06T03:59:16.157 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.155+0000 7feb6ab65700 1 -- 172.21.15.152:0/3748116389 shutdown_connections 2023-12-06T03:59:16.157 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.155+0000 7feb6ab65700 1 --2- 172.21.15.152:0/3748116389 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb640f5410 0x7feb640f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:16.157 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.155+0000 7feb6ab65700 1 -- 172.21.15.152:0/3748116389 >> 172.21.15.152:0/3748116389 conn(0x7feb640f0b70 msgr2=0x7feb640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:16.157 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.155+0000 7feb6ab65700 1 -- 172.21.15.152:0/3748116389 shutdown_connections 2023-12-06T03:59:16.157 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.155+0000 7feb6ab65700 1 -- 172.21.15.152:0/3748116389 wait complete. 2023-12-06T03:59:16.157 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.156+0000 7feb6ab65700 1 Processor -- start 2023-12-06T03:59:16.157 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.156+0000 7feb6ab65700 1 -- start start 2023-12-06T03:59:16.159 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.156+0000 7feb6ab65700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb640f5410 0x7feb64106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:16.159 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.156+0000 7feb6ab65700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb64106c20 con 0x7feb640f5410 2023-12-06T03:59:16.159 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.156+0000 7feb68901700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb640f5410 0x7feb64106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:16.159 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.156+0000 7feb68901700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb640f5410 0x7feb64106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:34196/0 (socket says 172.21.15.152:34196) 2023-12-06T03:59:16.159 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.156+0000 7feb68901700 1 -- 172.21.15.152:0/3581069918 learned_addr learned my addr 172.21.15.152:0/3581069918 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:16.159 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.157+0000 7feb68901700 1 -- 172.21.15.152:0/3581069918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb58005040 con 0x7feb640f5410 2023-12-06T03:59:16.159 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.157+0000 7feb68901700 1 --2- 172.21.15.152:0/3581069918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb640f5410 0x7feb64106730 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7feb58013510 tx=0x7feb5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:16.159 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.157+0000 7feb617fa700 1 -- 172.21.15.152:0/3581069918 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb58009320 con 0x7feb640f5410 2023-12-06T03:59:16.159 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.157+0000 7feb617fa700 1 -- 172.21.15.152:0/3581069918 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb58013f00 con 0x7feb640f5410 2023-12-06T03:59:16.159 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.157+0000 7feb6ab65700 1 -- 172.21.15.152:0/3581069918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb64106e20 con 0x7feb640f5410 2023-12-06T03:59:16.159 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.157+0000 7feb617fa700 1 -- 172.21.15.152:0/3581069918 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb58014d90 con 0x7feb640f5410 2023-12-06T03:59:16.159 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.157+0000 7feb6ab65700 1 -- 172.21.15.152:0/3581069918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb641072b0 con 0x7feb640f5410 2023-12-06T03:59:16.159 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.158+0000 7feb617fa700 1 -- 172.21.15.152:0/3581069918 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb58014420 con 0x7feb640f5410 2023-12-06T03:59:16.159 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.158+0000 7feb6ab65700 1 -- 172.21.15.152:0/3581069918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb64045ad0 con 0x7feb640f5410 2023-12-06T03:59:16.160 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.158+0000 7feb617fa700 1 --2- 172.21.15.152:0/3581069918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb5402ff60 0x7feb54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:16.160 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.158+0000 7feb617fa700 1 -- 172.21.15.152:0/3581069918 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb58041070 con 0x7feb640f5410 2023-12-06T03:59:16.160 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.158+0000 7feb63fff700 1 --2- 172.21.15.152:0/3581069918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb5402ff60 0x7feb54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:16.160 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.159+0000 7feb63fff700 1 --2- 172.21.15.152:0/3581069918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb5402ff60 0x7feb54032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7feb500096a0 tx=0x7feb50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:16.163 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.163+0000 7feb617fa700 1 -- 172.21.15.152:0/3581069918 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb580095f0 con 0x7feb640f5410 2023-12-06T03:59:16.369 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.368+0000 7feb6ab65700 1 -- 172.21.15.152:0/3581069918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb640fae60 con 0x7feb640f5410 2023-12-06T03:59:16.371 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.369+0000 7feb617fa700 1 -- 172.21.15.152:0/3581069918 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feb5801db70 con 0x7feb640f5410 2023-12-06T03:59:16.371 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:16.371 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:16.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.370+0000 7feb6ab65700 1 -- 172.21.15.152:0/3581069918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb5402ff60 msgr2=0x7feb54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:16.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.370+0000 7feb6ab65700 1 --2- 172.21.15.152:0/3581069918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb5402ff60 0x7feb54032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7feb500096a0 tx=0x7feb50006b40).stop 2023-12-06T03:59:16.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.370+0000 7feb6ab65700 1 -- 172.21.15.152:0/3581069918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb640f5410 msgr2=0x7feb64106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:16.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.370+0000 7feb6ab65700 1 --2- 172.21.15.152:0/3581069918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb640f5410 0x7feb64106730 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7feb58013510 tx=0x7feb5800b040).stop 2023-12-06T03:59:16.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.370+0000 7feb6ab65700 1 -- 172.21.15.152:0/3581069918 shutdown_connections 2023-12-06T03:59:16.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.370+0000 7feb6ab65700 1 --2- 172.21.15.152:0/3581069918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7feb5402ff60 0x7feb54032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:16.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.370+0000 7feb6ab65700 1 --2- 172.21.15.152:0/3581069918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7feb640f5410 0x7feb64106730 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:16.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.370+0000 7feb6ab65700 1 -- 172.21.15.152:0/3581069918 >> 172.21.15.152:0/3581069918 conn(0x7feb640f0b70 msgr2=0x7feb640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:16.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.371+0000 7feb6ab65700 1 -- 172.21.15.152:0/3581069918 shutdown_connections 2023-12-06T03:59:16.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:16.371+0000 7feb6ab65700 1 -- 172.21.15.152:0/3581069918 wait complete. 2023-12-06T03:59:16.373 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:16.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:16 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:16.369129+0000 mon.smithi132 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.152:0/3581069918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:17.779 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:17.779 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:17.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:17 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:16.444611+0000 mgr.smithi132.gokxja (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:18.901 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.899+0000 7f515a326700 1 -- 172.21.15.152:0/852351779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51540f47c0 msgr2=0x7f51540f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:18.901 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.899+0000 7f515a326700 1 --2- 172.21.15.152:0/852351779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51540f47c0 0x7f51540f4be0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f5148009fb0 tx=0x7f5148009b30).stop 2023-12-06T03:59:18.901 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.899+0000 7f515a326700 1 -- 172.21.15.152:0/852351779 shutdown_connections 2023-12-06T03:59:18.901 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.899+0000 7f515a326700 1 --2- 172.21.15.152:0/852351779 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51540f47c0 0x7f51540f4be0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:18.901 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.899+0000 7f515a326700 1 -- 172.21.15.152:0/852351779 >> 172.21.15.152:0/852351779 conn(0x7f51540f02e0 msgr2=0x7f51540f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:18.901 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.899+0000 7f515a326700 1 -- 172.21.15.152:0/852351779 shutdown_connections 2023-12-06T03:59:18.901 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.899+0000 7f515a326700 1 -- 172.21.15.152:0/852351779 wait complete. 2023-12-06T03:59:18.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.900+0000 7f515a326700 1 Processor -- start 2023-12-06T03:59:18.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.900+0000 7f515a326700 1 -- start start 2023-12-06T03:59:18.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.900+0000 7f515a326700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51540f47c0 0x7f5154106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:18.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.900+0000 7f515a326700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5154106bf0 con 0x7f51540f47c0 2023-12-06T03:59:18.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.900+0000 7f5153fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51540f47c0 0x7f5154106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:18.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.900+0000 7f5153fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51540f47c0 0x7f5154106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:34218/0 (socket says 172.21.15.152:34218) 2023-12-06T03:59:18.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.900+0000 7f5153fff700 1 -- 172.21.15.152:0/1449988918 learned_addr learned my addr 172.21.15.152:0/1449988918 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:18.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.901+0000 7f5153fff700 1 -- 172.21.15.152:0/1449988918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5148005040 con 0x7f51540f47c0 2023-12-06T03:59:18.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.901+0000 7f5153fff700 1 --2- 172.21.15.152:0/1449988918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51540f47c0 0x7f5154106700 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f5148013510 tx=0x7f514800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:18.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.901+0000 7f5150ff9700 1 -- 172.21.15.152:0/1449988918 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5148009320 con 0x7f51540f47c0 2023-12-06T03:59:18.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.901+0000 7f515a326700 1 -- 172.21.15.152:0/1449988918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5154106df0 con 0x7f51540f47c0 2023-12-06T03:59:18.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.901+0000 7f5150ff9700 1 -- 172.21.15.152:0/1449988918 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5148013f00 con 0x7f51540f47c0 2023-12-06T03:59:18.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.901+0000 7f5150ff9700 1 -- 172.21.15.152:0/1449988918 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5148014d90 con 0x7f51540f47c0 2023-12-06T03:59:18.906 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.901+0000 7f515a326700 1 -- 172.21.15.152:0/1449988918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5154107280 con 0x7f51540f47c0 2023-12-06T03:59:18.906 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.902+0000 7f5150ff9700 1 -- 172.21.15.152:0/1449988918 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51480095f0 con 0x7f51540f47c0 2023-12-06T03:59:18.906 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.902+0000 7f515a326700 1 -- 172.21.15.152:0/1449988918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5154045ad0 con 0x7f51540f47c0 2023-12-06T03:59:18.906 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.902+0000 7f5150ff9700 1 --2- 172.21.15.152:0/1449988918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f513c02ff60 0x7f513c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:18.906 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.902+0000 7f5150ff9700 1 -- 172.21.15.152:0/1449988918 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f51480410c0 con 0x7f51540f47c0 2023-12-06T03:59:18.906 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.902+0000 7f51537fe700 1 --2- 172.21.15.152:0/1449988918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f513c02ff60 0x7f513c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:18.906 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.903+0000 7f51537fe700 1 --2- 172.21.15.152:0/1449988918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f513c02ff60 0x7f513c032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f51440096a0 tx=0x7f5144006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:18.907 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:18.907+0000 7f5150ff9700 1 -- 172.21.15.152:0/1449988918 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5148014420 con 0x7f51540f47c0 2023-12-06T03:59:19.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:19.111+0000 7f515a326700 1 -- 172.21.15.152:0/1449988918 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5154108130 con 0x7f51540f47c0 2023-12-06T03:59:19.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:19.111+0000 7f5150ff9700 1 -- 172.21.15.152:0/1449988918 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5148010230 con 0x7f51540f47c0 2023-12-06T03:59:19.113 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:19.113 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:19.114 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:19.113+0000 7f515a326700 1 -- 172.21.15.152:0/1449988918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f513c02ff60 msgr2=0x7f513c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:19.114 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:19.113+0000 7f515a326700 1 --2- 172.21.15.152:0/1449988918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f513c02ff60 0x7f513c032420 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f51440096a0 tx=0x7f5144006b40).stop 2023-12-06T03:59:19.114 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:19.113+0000 7f515a326700 1 -- 172.21.15.152:0/1449988918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51540f47c0 msgr2=0x7f5154106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:19.114 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:19.113+0000 7f515a326700 1 --2- 172.21.15.152:0/1449988918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51540f47c0 0x7f5154106700 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f5148013510 tx=0x7f514800b040).stop 2023-12-06T03:59:19.114 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:19.113+0000 7f515a326700 1 -- 172.21.15.152:0/1449988918 shutdown_connections 2023-12-06T03:59:19.114 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:19.113+0000 7f515a326700 1 --2- 172.21.15.152:0/1449988918 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f513c02ff60 0x7f513c032420 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:19.114 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:19.113+0000 7f515a326700 1 --2- 172.21.15.152:0/1449988918 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f51540f47c0 0x7f5154106700 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:19.114 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:19.113+0000 7f515a326700 1 -- 172.21.15.152:0/1449988918 >> 172.21.15.152:0/1449988918 conn(0x7f51540f02e0 msgr2=0x7f51540f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:19.114 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:19.113+0000 7f515a326700 1 -- 172.21.15.152:0/1449988918 shutdown_connections 2023-12-06T03:59:19.115 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:19.113+0000 7f515a326700 1 -- 172.21.15.152:0/1449988918 wait complete. 2023-12-06T03:59:19.115 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:19.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:19 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:18.444938+0000 mgr.smithi132.gokxja (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:19.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:19 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:19.111917+0000 mon.smithi132 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.152:0/1449988918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:20.492 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:20.492 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:21.522 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.520+0000 7f5c6eab5700 1 -- 172.21.15.152:0/397708293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c680f6510 msgr2=0x7f5c680f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:21.522 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.520+0000 7f5c6eab5700 1 --2- 172.21.15.152:0/397708293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c680f6510 0x7f5c680f6930 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f5c5c009fb0 tx=0x7f5c5c009b30).stop 2023-12-06T03:59:21.522 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.520+0000 7f5c6eab5700 1 -- 172.21.15.152:0/397708293 shutdown_connections 2023-12-06T03:59:21.522 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.520+0000 7f5c6eab5700 1 --2- 172.21.15.152:0/397708293 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c680f6510 0x7f5c680f6930 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:21.523 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.520+0000 7f5c6eab5700 1 -- 172.21.15.152:0/397708293 >> 172.21.15.152:0/397708293 conn(0x7f5c680f1c70 msgr2=0x7f5c680f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:21.523 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.520+0000 7f5c6eab5700 1 -- 172.21.15.152:0/397708293 shutdown_connections 2023-12-06T03:59:21.523 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.520+0000 7f5c6eab5700 1 -- 172.21.15.152:0/397708293 wait complete. 2023-12-06T03:59:21.523 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.521+0000 7f5c6eab5700 1 Processor -- start 2023-12-06T03:59:21.523 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.521+0000 7f5c6eab5700 1 -- start start 2023-12-06T03:59:21.524 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.521+0000 7f5c6eab5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c680f6510 0x7f5c68106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:21.525 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.521+0000 7f5c6eab5700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c68106c20 con 0x7f5c680f6510 2023-12-06T03:59:21.525 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.521+0000 7f5c6c851700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c680f6510 0x7f5c68106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:21.525 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.521+0000 7f5c6c851700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c680f6510 0x7f5c68106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:60640/0 (socket says 172.21.15.152:60640) 2023-12-06T03:59:21.525 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.521+0000 7f5c6c851700 1 -- 172.21.15.152:0/2252780126 learned_addr learned my addr 172.21.15.152:0/2252780126 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:21.525 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.522+0000 7f5c6c851700 1 -- 172.21.15.152:0/2252780126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c005040 con 0x7f5c680f6510 2023-12-06T03:59:21.525 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.522+0000 7f5c6c851700 1 --2- 172.21.15.152:0/2252780126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c680f6510 0x7f5c68106730 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f5c5c014000 tx=0x7f5c5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:21.525 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.522+0000 7f5c657fa700 1 -- 172.21.15.152:0/2252780126 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c5c0095f0 con 0x7f5c680f6510 2023-12-06T03:59:21.525 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.522+0000 7f5c6eab5700 1 -- 172.21.15.152:0/2252780126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c68106e20 con 0x7f5c680f6510 2023-12-06T03:59:21.525 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.522+0000 7f5c657fa700 1 -- 172.21.15.152:0/2252780126 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c5c007e70 con 0x7f5c680f6510 2023-12-06T03:59:21.525 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.522+0000 7f5c657fa700 1 -- 172.21.15.152:0/2252780126 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c5c014ea0 con 0x7f5c680f6510 2023-12-06T03:59:21.525 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.522+0000 7f5c6eab5700 1 -- 172.21.15.152:0/2252780126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c681072b0 con 0x7f5c680f6510 2023-12-06T03:59:21.525 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.523+0000 7f5c657fa700 1 -- 172.21.15.152:0/2252780126 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c5c014690 con 0x7f5c680f6510 2023-12-06T03:59:21.525 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.523+0000 7f5c6eab5700 1 -- 172.21.15.152:0/2252780126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c68045ad0 con 0x7f5c680f6510 2023-12-06T03:59:21.526 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.523+0000 7f5c657fa700 1 --2- 172.21.15.152:0/2252780126 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5c5802ff60 0x7f5c58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:21.526 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.523+0000 7f5c657fa700 1 -- 172.21.15.152:0/2252780126 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c5c040930 con 0x7f5c680f6510 2023-12-06T03:59:21.526 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.523+0000 7f5c67fff700 1 --2- 172.21.15.152:0/2252780126 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5c5802ff60 0x7f5c58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:21.526 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.524+0000 7f5c67fff700 1 --2- 172.21.15.152:0/2252780126 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5c5802ff60 0x7f5c58032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f5c540096a0 tx=0x7f5c54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:21.529 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.528+0000 7f5c657fa700 1 -- 172.21.15.152:0/2252780126 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c5c00f300 con 0x7f5c680f6510 2023-12-06T03:59:21.737 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.736+0000 7f5c6eab5700 1 -- 172.21.15.152:0/2252780126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c680fbf60 con 0x7f5c680f6510 2023-12-06T03:59:21.738 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.737+0000 7f5c657fa700 1 -- 172.21.15.152:0/2252780126 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5c5c00f300 con 0x7f5c680f6510 2023-12-06T03:59:21.738 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:21.739 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:21.741 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.738+0000 7f5c6eab5700 1 -- 172.21.15.152:0/2252780126 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5c5802ff60 msgr2=0x7f5c58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:21.741 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.738+0000 7f5c6eab5700 1 --2- 172.21.15.152:0/2252780126 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5c5802ff60 0x7f5c58032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f5c540096a0 tx=0x7f5c54006b40).stop 2023-12-06T03:59:21.741 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.738+0000 7f5c6eab5700 1 -- 172.21.15.152:0/2252780126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c680f6510 msgr2=0x7f5c68106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:21.741 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.738+0000 7f5c6eab5700 1 --2- 172.21.15.152:0/2252780126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c680f6510 0x7f5c68106730 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f5c5c014000 tx=0x7f5c5c00b040).stop 2023-12-06T03:59:21.742 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.738+0000 7f5c6eab5700 1 -- 172.21.15.152:0/2252780126 shutdown_connections 2023-12-06T03:59:21.742 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.738+0000 7f5c6eab5700 1 --2- 172.21.15.152:0/2252780126 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5c5802ff60 0x7f5c58032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:21.742 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.738+0000 7f5c6eab5700 1 --2- 172.21.15.152:0/2252780126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5c680f6510 0x7f5c68106730 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:21.742 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.738+0000 7f5c6eab5700 1 -- 172.21.15.152:0/2252780126 >> 172.21.15.152:0/2252780126 conn(0x7f5c680f1c70 msgr2=0x7f5c680f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:21.742 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.739+0000 7f5c6eab5700 1 -- 172.21.15.152:0/2252780126 shutdown_connections 2023-12-06T03:59:21.742 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:21.739+0000 7f5c6eab5700 1 -- 172.21.15.152:0/2252780126 wait complete. 2023-12-06T03:59:21.742 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:21.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:20.445276+0000 mgr.smithi132.gokxja (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:22.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:22 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:21.737925+0000 mon.smithi132 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.152:0/2252780126' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:23.095 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:23.095 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:23.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:22.445621+0000 mgr.smithi132.gokxja (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:24.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.178+0000 7fbca1826700 1 -- 172.21.15.152:0/222067612 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc9c0f5410 msgr2=0x7fbc9c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:24.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.178+0000 7fbca1826700 1 --2- 172.21.15.152:0/222067612 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc9c0f5410 0x7fbc9c0f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7fbc90009fb0 tx=0x7fbc90009b30).stop 2023-12-06T03:59:24.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.178+0000 7fbca1826700 1 -- 172.21.15.152:0/222067612 shutdown_connections 2023-12-06T03:59:24.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.178+0000 7fbca1826700 1 --2- 172.21.15.152:0/222067612 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc9c0f5410 0x7fbc9c0f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:24.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.178+0000 7fbca1826700 1 -- 172.21.15.152:0/222067612 >> 172.21.15.152:0/222067612 conn(0x7fbc9c0f0b70 msgr2=0x7fbc9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:24.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.178+0000 7fbca1826700 1 -- 172.21.15.152:0/222067612 shutdown_connections 2023-12-06T03:59:24.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.178+0000 7fbca1826700 1 -- 172.21.15.152:0/222067612 wait complete. 2023-12-06T03:59:24.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.179+0000 7fbca1826700 1 Processor -- start 2023-12-06T03:59:24.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.179+0000 7fbca1826700 1 -- start start 2023-12-06T03:59:24.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.179+0000 7fbca1826700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc9c0f5410 0x7fbc9c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:24.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.179+0000 7fbca1826700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc9c105b20 con 0x7fbc9c0f5410 2023-12-06T03:59:24.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.179+0000 7fbc9affd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc9c0f5410 0x7fbc9c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:24.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.179+0000 7fbc9affd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc9c0f5410 0x7fbc9c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:60658/0 (socket says 172.21.15.152:60658) 2023-12-06T03:59:24.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.179+0000 7fbc9affd700 1 -- 172.21.15.152:0/6723740 learned_addr learned my addr 172.21.15.152:0/6723740 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:24.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.180+0000 7fbc9affd700 1 -- 172.21.15.152:0/6723740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc90005040 con 0x7fbc9c0f5410 2023-12-06T03:59:24.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.180+0000 7fbc9affd700 1 --2- 172.21.15.152:0/6723740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc9c0f5410 0x7fbc9c105630 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fbc90014000 tx=0x7fbc9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:24.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.180+0000 7fbc837fe700 1 -- 172.21.15.152:0/6723740 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc900095f0 con 0x7fbc9c0f5410 2023-12-06T03:59:24.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.180+0000 7fbca1826700 1 -- 172.21.15.152:0/6723740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc9c105d20 con 0x7fbc9c0f5410 2023-12-06T03:59:24.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.180+0000 7fbc837fe700 1 -- 172.21.15.152:0/6723740 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc90007e70 con 0x7fbc9c0f5410 2023-12-06T03:59:24.183 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.180+0000 7fbc837fe700 1 -- 172.21.15.152:0/6723740 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc90014ea0 con 0x7fbc9c0f5410 2023-12-06T03:59:24.183 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.180+0000 7fbca1826700 1 -- 172.21.15.152:0/6723740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc9c1061b0 con 0x7fbc9c0f5410 2023-12-06T03:59:24.183 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.181+0000 7fbca1826700 1 -- 172.21.15.152:0/6723740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc9c045ad0 con 0x7fbc9c0f5410 2023-12-06T03:59:24.183 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.181+0000 7fbc837fe700 1 -- 172.21.15.152:0/6723740 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc90025070 con 0x7fbc9c0f5410 2023-12-06T03:59:24.184 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.181+0000 7fbc837fe700 1 --2- 172.21.15.152:0/6723740 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbc8402ff60 0x7fbc84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:24.184 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.181+0000 7fbc837fe700 1 -- 172.21.15.152:0/6723740 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc900409c0 con 0x7fbc9c0f5410 2023-12-06T03:59:24.184 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.181+0000 7fbc9a7fc700 1 --2- 172.21.15.152:0/6723740 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbc8402ff60 0x7fbc84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:24.184 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.182+0000 7fbc9a7fc700 1 --2- 172.21.15.152:0/6723740 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbc8402ff60 0x7fbc84032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fbc8c0096a0 tx=0x7fbc8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:24.187 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.186+0000 7fbc837fe700 1 -- 172.21.15.152:0/6723740 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc900079c0 con 0x7fbc9c0f5410 2023-12-06T03:59:24.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.394+0000 7fbca1826700 1 -- 172.21.15.152:0/6723740 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc9c0fae60 con 0x7fbc9c0f5410 2023-12-06T03:59:24.396 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.394+0000 7fbc837fe700 1 -- 172.21.15.152:0/6723740 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbc9000f300 con 0x7fbc9c0f5410 2023-12-06T03:59:24.396 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:24.396 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:24.398 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.396+0000 7fbca1826700 1 -- 172.21.15.152:0/6723740 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbc8402ff60 msgr2=0x7fbc84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:24.398 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.396+0000 7fbca1826700 1 --2- 172.21.15.152:0/6723740 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbc8402ff60 0x7fbc84032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fbc8c0096a0 tx=0x7fbc8c006b40).stop 2023-12-06T03:59:24.398 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.396+0000 7fbca1826700 1 -- 172.21.15.152:0/6723740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc9c0f5410 msgr2=0x7fbc9c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:24.398 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.396+0000 7fbca1826700 1 --2- 172.21.15.152:0/6723740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc9c0f5410 0x7fbc9c105630 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fbc90014000 tx=0x7fbc9000b040).stop 2023-12-06T03:59:24.398 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.396+0000 7fbca1826700 1 -- 172.21.15.152:0/6723740 shutdown_connections 2023-12-06T03:59:24.398 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.396+0000 7fbca1826700 1 --2- 172.21.15.152:0/6723740 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbc8402ff60 0x7fbc84032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:24.398 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.396+0000 7fbca1826700 1 --2- 172.21.15.152:0/6723740 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbc9c0f5410 0x7fbc9c105630 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:24.398 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.396+0000 7fbca1826700 1 -- 172.21.15.152:0/6723740 >> 172.21.15.152:0/6723740 conn(0x7fbc9c0f0b70 msgr2=0x7fbc9c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:24.398 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.396+0000 7fbca1826700 1 -- 172.21.15.152:0/6723740 shutdown_connections 2023-12-06T03:59:24.398 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:24.396+0000 7fbca1826700 1 -- 172.21.15.152:0/6723740 wait complete. 2023-12-06T03:59:24.399 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:24.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:24 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:24.395761+0000 mon.smithi132 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.152:0/6723740' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:25.760 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:25.761 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:25.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:24.445944+0000 mgr.smithi132.gokxja (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:26.867 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.865+0000 7f675a35a700 1 -- 172.21.15.152:0/4085771807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6754060200 msgr2=0x7f67540f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:26.868 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.865+0000 7f675a35a700 1 --2- 172.21.15.152:0/4085771807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6754060200 0x7f67540f9170 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f6744009fb0 tx=0x7f6744009b30).stop 2023-12-06T03:59:26.868 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.865+0000 7f675a35a700 1 -- 172.21.15.152:0/4085771807 shutdown_connections 2023-12-06T03:59:26.868 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.865+0000 7f675a35a700 1 --2- 172.21.15.152:0/4085771807 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6754060200 0x7f67540f9170 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:26.868 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.865+0000 7f675a35a700 1 -- 172.21.15.152:0/4085771807 >> 172.21.15.152:0/4085771807 conn(0x7f67540f0b70 msgr2=0x7f67540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:26.868 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.865+0000 7f675a35a700 1 -- 172.21.15.152:0/4085771807 shutdown_connections 2023-12-06T03:59:26.868 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.865+0000 7f675a35a700 1 -- 172.21.15.152:0/4085771807 wait complete. 2023-12-06T03:59:26.868 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.866+0000 7f675a35a700 1 Processor -- start 2023-12-06T03:59:26.868 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.866+0000 7f675a35a700 1 -- start start 2023-12-06T03:59:26.868 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.866+0000 7f675a35a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6754060200 0x7f67541066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:26.868 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.866+0000 7f675a35a700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6754106b90 con 0x7f6754060200 2023-12-06T03:59:26.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.866+0000 7f6753fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6754060200 0x7f67541066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:26.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.866+0000 7f6753fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6754060200 0x7f67541066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:60672/0 (socket says 172.21.15.152:60672) 2023-12-06T03:59:26.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.866+0000 7f6753fff700 1 -- 172.21.15.152:0/2956660101 learned_addr learned my addr 172.21.15.152:0/2956660101 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:26.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.867+0000 7f6753fff700 1 -- 172.21.15.152:0/2956660101 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6744005040 con 0x7f6754060200 2023-12-06T03:59:26.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.867+0000 7f6753fff700 1 --2- 172.21.15.152:0/2956660101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6754060200 0x7f67541066a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f6744009830 tx=0x7f674400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:26.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.867+0000 7f6750ff9700 1 -- 172.21.15.152:0/2956660101 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6744009320 con 0x7f6754060200 2023-12-06T03:59:26.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.867+0000 7f675a35a700 1 -- 172.21.15.152:0/2956660101 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6754106d90 con 0x7f6754060200 2023-12-06T03:59:26.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.867+0000 7f6750ff9700 1 -- 172.21.15.152:0/2956660101 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67440136c0 con 0x7f6754060200 2023-12-06T03:59:26.870 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.867+0000 7f6750ff9700 1 -- 172.21.15.152:0/2956660101 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f674401ee90 con 0x7f6754060200 2023-12-06T03:59:26.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.867+0000 7f675a35a700 1 -- 172.21.15.152:0/2956660101 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6754107220 con 0x7f6754060200 2023-12-06T03:59:26.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.868+0000 7f6750ff9700 1 -- 172.21.15.152:0/2956660101 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6744014440 con 0x7f6754060200 2023-12-06T03:59:26.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.868+0000 7f6750ff9700 1 --2- 172.21.15.152:0/2956660101 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f673c02ff10 0x7f673c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:26.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.868+0000 7f6750ff9700 1 -- 172.21.15.152:0/2956660101 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6744041070 con 0x7f6754060200 2023-12-06T03:59:26.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.868+0000 7f675a35a700 1 -- 172.21.15.152:0/2956660101 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6754045ad0 con 0x7f6754060200 2023-12-06T03:59:26.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.868+0000 7f67537fe700 1 --2- 172.21.15.152:0/2956660101 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f673c02ff10 0x7f673c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:26.871 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.868+0000 7f67537fe700 1 --2- 172.21.15.152:0/2956660101 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f673c02ff10 0x7f673c0323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f67480096a0 tx=0x7f6748006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:26.874 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:26.873+0000 7f6750ff9700 1 -- 172.21.15.152:0/2956660101 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67440273f0 con 0x7f6754060200 2023-12-06T03:59:27.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:27.078+0000 7f675a35a700 1 -- 172.21.15.152:0/2956660101 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6754107f40 con 0x7f6754060200 2023-12-06T03:59:27.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:27.078+0000 7f6750ff9700 1 -- 172.21.15.152:0/2956660101 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f674401c320 con 0x7f6754060200 2023-12-06T03:59:27.081 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:27.081 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:27.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:27.080+0000 7f675a35a700 1 -- 172.21.15.152:0/2956660101 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f673c02ff10 msgr2=0x7f673c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:27.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:27.080+0000 7f675a35a700 1 --2- 172.21.15.152:0/2956660101 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f673c02ff10 0x7f673c0323d0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f67480096a0 tx=0x7f6748006b40).stop 2023-12-06T03:59:27.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:27.080+0000 7f675a35a700 1 -- 172.21.15.152:0/2956660101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6754060200 msgr2=0x7f67541066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:27.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:27.080+0000 7f675a35a700 1 --2- 172.21.15.152:0/2956660101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6754060200 0x7f67541066a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f6744009830 tx=0x7f674400b040).stop 2023-12-06T03:59:27.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:27.080+0000 7f675a35a700 1 -- 172.21.15.152:0/2956660101 shutdown_connections 2023-12-06T03:59:27.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:27.080+0000 7f675a35a700 1 --2- 172.21.15.152:0/2956660101 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f673c02ff10 0x7f673c0323d0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:27.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:27.080+0000 7f675a35a700 1 --2- 172.21.15.152:0/2956660101 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6754060200 0x7f67541066a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:27.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:27.080+0000 7f675a35a700 1 -- 172.21.15.152:0/2956660101 >> 172.21.15.152:0/2956660101 conn(0x7f67540f0b70 msgr2=0x7f67540f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:27.082 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:27.080+0000 7f675a35a700 1 -- 172.21.15.152:0/2956660101 shutdown_connections 2023-12-06T03:59:27.083 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:27.080+0000 7f675a35a700 1 -- 172.21.15.152:0/2956660101 wait complete. 2023-12-06T03:59:27.083 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:27.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:26.446212+0000 mgr.smithi132.gokxja (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:27.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:27.080127+0000 mon.smithi132 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.152:0/2956660101' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:28.452 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:28.452 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:29.599 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.597+0000 7ff1a9a37700 1 -- 172.21.15.152:0/1693178257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a40f6510 msgr2=0x7ff1a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:29.599 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.597+0000 7ff1a9a37700 1 --2- 172.21.15.152:0/1693178257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a40f6510 0x7ff1a40f6930 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7ff198009fb0 tx=0x7ff198009b30).stop 2023-12-06T03:59:29.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.597+0000 7ff1a9a37700 1 -- 172.21.15.152:0/1693178257 shutdown_connections 2023-12-06T03:59:29.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.597+0000 7ff1a9a37700 1 --2- 172.21.15.152:0/1693178257 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a40f6510 0x7ff1a40f6930 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:29.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.597+0000 7ff1a9a37700 1 -- 172.21.15.152:0/1693178257 >> 172.21.15.152:0/1693178257 conn(0x7ff1a40f1c70 msgr2=0x7ff1a40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:29.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.598+0000 7ff1a9a37700 1 -- 172.21.15.152:0/1693178257 shutdown_connections 2023-12-06T03:59:29.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.598+0000 7ff1a9a37700 1 -- 172.21.15.152:0/1693178257 wait complete. 2023-12-06T03:59:29.601 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.598+0000 7ff1a9a37700 1 Processor -- start 2023-12-06T03:59:29.601 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.598+0000 7ff1a9a37700 1 -- start start 2023-12-06T03:59:29.601 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.598+0000 7ff1a9a37700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a40f6510 0x7ff1a4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:29.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.599+0000 7ff1a9a37700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1a4106c20 con 0x7ff1a40f6510 2023-12-06T03:59:29.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.599+0000 7ff1a2ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a40f6510 0x7ff1a4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:29.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.599+0000 7ff1a2ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a40f6510 0x7ff1a4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:36540/0 (socket says 172.21.15.152:36540) 2023-12-06T03:59:29.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.599+0000 7ff1a2ffd700 1 -- 172.21.15.152:0/268161697 learned_addr learned my addr 172.21.15.152:0/268161697 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:29.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.599+0000 7ff1a2ffd700 1 -- 172.21.15.152:0/268161697 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff198005040 con 0x7ff1a40f6510 2023-12-06T03:59:29.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.599+0000 7ff1a2ffd700 1 --2- 172.21.15.152:0/268161697 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a40f6510 0x7ff1a4106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7ff198014000 tx=0x7ff19800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:29.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.599+0000 7ff18b7fe700 1 -- 172.21.15.152:0/268161697 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff1980095f0 con 0x7ff1a40f6510 2023-12-06T03:59:29.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.599+0000 7ff1a9a37700 1 -- 172.21.15.152:0/268161697 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1a4106e20 con 0x7ff1a40f6510 2023-12-06T03:59:29.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.599+0000 7ff18b7fe700 1 -- 172.21.15.152:0/268161697 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff198007e70 con 0x7ff1a40f6510 2023-12-06T03:59:29.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.599+0000 7ff18b7fe700 1 -- 172.21.15.152:0/268161697 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff198014ea0 con 0x7ff1a40f6510 2023-12-06T03:59:29.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.599+0000 7ff1a9a37700 1 -- 172.21.15.152:0/268161697 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1a41072b0 con 0x7ff1a40f6510 2023-12-06T03:59:29.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.600+0000 7ff18b7fe700 1 -- 172.21.15.152:0/268161697 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff198025070 con 0x7ff1a40f6510 2023-12-06T03:59:29.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.600+0000 7ff1a9a37700 1 -- 172.21.15.152:0/268161697 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1a4045ad0 con 0x7ff1a40f6510 2023-12-06T03:59:29.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.600+0000 7ff18b7fe700 1 --2- 172.21.15.152:0/268161697 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff18c030000 0x7ff18c0324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:29.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.600+0000 7ff18b7fe700 1 -- 172.21.15.152:0/268161697 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff1980409e0 con 0x7ff1a40f6510 2023-12-06T03:59:29.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.600+0000 7ff1a27fc700 1 --2- 172.21.15.152:0/268161697 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff18c030000 0x7ff18c0324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:29.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.601+0000 7ff1a27fc700 1 --2- 172.21.15.152:0/268161697 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff18c030000 0x7ff18c0324c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff1940096a0 tx=0x7ff194006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:29.606 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.605+0000 7ff18b7fe700 1 -- 172.21.15.152:0/268161697 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff19800f4e0 con 0x7ff1a40f6510 2023-12-06T03:59:29.814 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.812+0000 7ff1a9a37700 1 -- 172.21.15.152:0/268161697 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff1a40fbf60 con 0x7ff1a40f6510 2023-12-06T03:59:29.814 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.813+0000 7ff18b7fe700 1 -- 172.21.15.152:0/268161697 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff19800f300 con 0x7ff1a40f6510 2023-12-06T03:59:29.815 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:29.815 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:29.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.814+0000 7ff1a9a37700 1 -- 172.21.15.152:0/268161697 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff18c030000 msgr2=0x7ff18c0324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:29.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.814+0000 7ff1a9a37700 1 --2- 172.21.15.152:0/268161697 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff18c030000 0x7ff18c0324c0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff1940096a0 tx=0x7ff194006b40).stop 2023-12-06T03:59:29.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.814+0000 7ff1a9a37700 1 -- 172.21.15.152:0/268161697 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a40f6510 msgr2=0x7ff1a4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:29.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.814+0000 7ff1a9a37700 1 --2- 172.21.15.152:0/268161697 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a40f6510 0x7ff1a4106730 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7ff198014000 tx=0x7ff19800b040).stop 2023-12-06T03:59:29.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.814+0000 7ff1a9a37700 1 -- 172.21.15.152:0/268161697 shutdown_connections 2023-12-06T03:59:29.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.814+0000 7ff1a9a37700 1 --2- 172.21.15.152:0/268161697 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff18c030000 0x7ff18c0324c0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:29.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.814+0000 7ff1a9a37700 1 --2- 172.21.15.152:0/268161697 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff1a40f6510 0x7ff1a4106730 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:29.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.814+0000 7ff1a9a37700 1 -- 172.21.15.152:0/268161697 >> 172.21.15.152:0/268161697 conn(0x7ff1a40f1c70 msgr2=0x7ff1a40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:29.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.814+0000 7ff1a9a37700 1 -- 172.21.15.152:0/268161697 shutdown_connections 2023-12-06T03:59:29.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:29.814+0000 7ff1a9a37700 1 -- 172.21.15.152:0/268161697 wait complete. 2023-12-06T03:59:29.817 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:29.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:28.446594+0000 mgr.smithi132.gokxja (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:30.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:30 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:29.814178+0000 mon.smithi132 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.152:0/268161697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:31.191 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:31.191 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:31.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:31 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:30.446897+0000 mgr.smithi132.gokxja (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:32.247 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.243+0000 7fe51a8c4700 1 -- 172.21.15.152:0/2501145273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5140f51a0 msgr2=0x7fe5140f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:32.247 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.243+0000 7fe51a8c4700 1 --2- 172.21.15.152:0/2501145273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5140f51a0 0x7fe5140f55c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fe510009fb0 tx=0x7fe510009b30).stop 2023-12-06T03:59:32.247 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.244+0000 7fe51a8c4700 1 -- 172.21.15.152:0/2501145273 shutdown_connections 2023-12-06T03:59:32.247 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.244+0000 7fe51a8c4700 1 --2- 172.21.15.152:0/2501145273 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5140f51a0 0x7fe5140f55c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:32.248 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.244+0000 7fe51a8c4700 1 -- 172.21.15.152:0/2501145273 >> 172.21.15.152:0/2501145273 conn(0x7fe5140f09a0 msgr2=0x7fe5140f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:32.248 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.244+0000 7fe51a8c4700 1 -- 172.21.15.152:0/2501145273 shutdown_connections 2023-12-06T03:59:32.248 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.244+0000 7fe51a8c4700 1 -- 172.21.15.152:0/2501145273 wait complete. 2023-12-06T03:59:32.248 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.244+0000 7fe51a8c4700 1 Processor -- start 2023-12-06T03:59:32.248 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.244+0000 7fe51a8c4700 1 -- start start 2023-12-06T03:59:32.248 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.244+0000 7fe51a8c4700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5140f51a0 0x7fe5141053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:32.248 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.244+0000 7fe51a8c4700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5141058a0 con 0x7fe5140f51a0 2023-12-06T03:59:32.248 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.245+0000 7fe5198c2700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5140f51a0 0x7fe5141053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:32.248 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.245+0000 7fe5198c2700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5140f51a0 0x7fe5141053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:36554/0 (socket says 172.21.15.152:36554) 2023-12-06T03:59:32.248 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.245+0000 7fe5198c2700 1 -- 172.21.15.152:0/1181310995 learned_addr learned my addr 172.21.15.152:0/1181310995 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:32.248 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.245+0000 7fe5198c2700 1 -- 172.21.15.152:0/1181310995 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe510005040 con 0x7fe5140f51a0 2023-12-06T03:59:32.248 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.245+0000 7fe5198c2700 1 --2- 172.21.15.152:0/1181310995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5140f51a0 0x7fe5141053b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fe510013510 tx=0x7fe51000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:32.248 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.245+0000 7fe50a7fc700 1 -- 172.21.15.152:0/1181310995 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe510009320 con 0x7fe5140f51a0 2023-12-06T03:59:32.248 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.245+0000 7fe51a8c4700 1 -- 172.21.15.152:0/1181310995 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe514105aa0 con 0x7fe5140f51a0 2023-12-06T03:59:32.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.245+0000 7fe50a7fc700 1 -- 172.21.15.152:0/1181310995 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe510013f00 con 0x7fe5140f51a0 2023-12-06T03:59:32.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.245+0000 7fe50a7fc700 1 -- 172.21.15.152:0/1181310995 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe510014dc0 con 0x7fe5140f51a0 2023-12-06T03:59:32.249 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.245+0000 7fe51a8c4700 1 -- 172.21.15.152:0/1181310995 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe514105f30 con 0x7fe5140f51a0 2023-12-06T03:59:32.250 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.246+0000 7fe50a7fc700 1 -- 172.21.15.152:0/1181310995 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe5100095f0 con 0x7fe5140f51a0 2023-12-06T03:59:32.250 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.246+0000 7fe51a8c4700 1 -- 172.21.15.152:0/1181310995 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe514045ad0 con 0x7fe5140f51a0 2023-12-06T03:59:32.250 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.246+0000 7fe50a7fc700 1 --2- 172.21.15.152:0/1181310995 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe50002ff60 0x7fe500032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:32.250 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.246+0000 7fe50a7fc700 1 -- 172.21.15.152:0/1181310995 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe510041100 con 0x7fe5140f51a0 2023-12-06T03:59:32.250 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.246+0000 7fe5190c1700 1 --2- 172.21.15.152:0/1181310995 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe50002ff60 0x7fe500032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:32.250 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.247+0000 7fe5190c1700 1 --2- 172.21.15.152:0/1181310995 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe50002ff60 0x7fe500032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fe5040096a0 tx=0x7fe504006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:32.253 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.251+0000 7fe50a7fc700 1 -- 172.21.15.152:0/1181310995 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe510014420 con 0x7fe5140f51a0 2023-12-06T03:59:32.460 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.459+0000 7fe51a8c4700 1 -- 172.21.15.152:0/1181310995 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe514106de0 con 0x7fe5140f51a0 2023-12-06T03:59:32.461 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.459+0000 7fe50a7fc700 1 -- 172.21.15.152:0/1181310995 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe510010230 con 0x7fe5140f51a0 2023-12-06T03:59:32.461 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:32.461 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:32.463 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.461+0000 7fe51a8c4700 1 -- 172.21.15.152:0/1181310995 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe50002ff60 msgr2=0x7fe500032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:32.463 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.461+0000 7fe51a8c4700 1 --2- 172.21.15.152:0/1181310995 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe50002ff60 0x7fe500032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fe5040096a0 tx=0x7fe504006b40).stop 2023-12-06T03:59:32.463 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.461+0000 7fe51a8c4700 1 -- 172.21.15.152:0/1181310995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5140f51a0 msgr2=0x7fe5141053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:32.463 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.461+0000 7fe51a8c4700 1 --2- 172.21.15.152:0/1181310995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5140f51a0 0x7fe5141053b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fe510013510 tx=0x7fe51000b040).stop 2023-12-06T03:59:32.463 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.461+0000 7fe51a8c4700 1 -- 172.21.15.152:0/1181310995 shutdown_connections 2023-12-06T03:59:32.463 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.461+0000 7fe51a8c4700 1 --2- 172.21.15.152:0/1181310995 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe50002ff60 0x7fe500032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:32.463 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.461+0000 7fe51a8c4700 1 --2- 172.21.15.152:0/1181310995 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe5140f51a0 0x7fe5141053b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:32.464 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.461+0000 7fe51a8c4700 1 -- 172.21.15.152:0/1181310995 >> 172.21.15.152:0/1181310995 conn(0x7fe5140f09a0 msgr2=0x7fe5140f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:32.464 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.461+0000 7fe51a8c4700 1 -- 172.21.15.152:0/1181310995 shutdown_connections 2023-12-06T03:59:32.464 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:32.461+0000 7fe51a8c4700 1 -- 172.21.15.152:0/1181310995 wait complete. 2023-12-06T03:59:32.464 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:32.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:32 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:32.461059+0000 mon.smithi132 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.152:0/1181310995' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:33.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:33 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:32.447261+0000 mgr.smithi132.gokxja (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:33.907 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:33.907 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:34.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.992+0000 7f471c3c9700 1 -- 172.21.15.152:0/1632000523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47140f5410 msgr2=0x7f47140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:34.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.992+0000 7f471c3c9700 1 --2- 172.21.15.152:0/1632000523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47140f5410 0x7f47140f5830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f4710009fb0 tx=0x7f4710009b30).stop 2023-12-06T03:59:34.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.992+0000 7f471c3c9700 1 -- 172.21.15.152:0/1632000523 shutdown_connections 2023-12-06T03:59:34.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.992+0000 7f471c3c9700 1 --2- 172.21.15.152:0/1632000523 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47140f5410 0x7f47140f5830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:34.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.992+0000 7f471c3c9700 1 -- 172.21.15.152:0/1632000523 >> 172.21.15.152:0/1632000523 conn(0x7f47140f0b70 msgr2=0x7f47140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:34.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.992+0000 7f471c3c9700 1 -- 172.21.15.152:0/1632000523 shutdown_connections 2023-12-06T03:59:34.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.992+0000 7f471c3c9700 1 -- 172.21.15.152:0/1632000523 wait complete. 2023-12-06T03:59:34.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.993+0000 7f471c3c9700 1 Processor -- start 2023-12-06T03:59:34.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.993+0000 7f471c3c9700 1 -- start start 2023-12-06T03:59:34.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.993+0000 7f471c3c9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47140f5410 0x7f4714109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:34.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.993+0000 7f471c3c9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4714060220 con 0x7f47140f5410 2023-12-06T03:59:34.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.993+0000 7f471a165700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47140f5410 0x7f4714109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:34.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.993+0000 7f471a165700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47140f5410 0x7f4714109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:36568/0 (socket says 172.21.15.152:36568) 2023-12-06T03:59:34.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.993+0000 7f471a165700 1 -- 172.21.15.152:0/1582828095 learned_addr learned my addr 172.21.15.152:0/1582828095 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:34.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.993+0000 7f471a165700 1 -- 172.21.15.152:0/1582828095 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4710005040 con 0x7f47140f5410 2023-12-06T03:59:34.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.993+0000 7f471a165700 1 --2- 172.21.15.152:0/1582828095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47140f5410 0x7f4714109450 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f4710014000 tx=0x7f471000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:34.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.994+0000 7f470affd700 1 -- 172.21.15.152:0/1582828095 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f47100095f0 con 0x7f47140f5410 2023-12-06T03:59:34.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.994+0000 7f470affd700 1 -- 172.21.15.152:0/1582828095 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4710007e70 con 0x7f47140f5410 2023-12-06T03:59:34.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.994+0000 7f471c3c9700 1 -- 172.21.15.152:0/1582828095 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4714060420 con 0x7f47140f5410 2023-12-06T03:59:34.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.994+0000 7f470affd700 1 -- 172.21.15.152:0/1582828095 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4710014ea0 con 0x7f47140f5410 2023-12-06T03:59:34.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.994+0000 7f471c3c9700 1 -- 172.21.15.152:0/1582828095 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4714109b00 con 0x7f47140f5410 2023-12-06T03:59:34.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.994+0000 7f470affd700 1 -- 172.21.15.152:0/1582828095 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4710025070 con 0x7f47140f5410 2023-12-06T03:59:34.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.995+0000 7f470affd700 1 --2- 172.21.15.152:0/1582828095 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f470002ff60 0x7f4700032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:34.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.995+0000 7f471c3c9700 1 -- 172.21.15.152:0/1582828095 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4714059100 con 0x7f47140f5410 2023-12-06T03:59:34.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.995+0000 7f470affd700 1 -- 172.21.15.152:0/1582828095 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f47100409f0 con 0x7f47140f5410 2023-12-06T03:59:34.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.995+0000 7f4719964700 1 --2- 172.21.15.152:0/1582828095 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f470002ff60 0x7f4700032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:34.998 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.995+0000 7f4719964700 1 --2- 172.21.15.152:0/1582828095 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f470002ff60 0x7f4700032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f47040096a0 tx=0x7f4704006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:35.007 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:34.999+0000 7f470affd700 1 -- 172.21.15.152:0/1582828095 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f471000f4e0 con 0x7f47140f5410 2023-12-06T03:59:35.208 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:35.206+0000 7f471c3c9700 1 -- 172.21.15.152:0/1582828095 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4714045ad0 con 0x7f47140f5410 2023-12-06T03:59:35.208 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:35.207+0000 7f470affd700 1 -- 172.21.15.152:0/1582828095 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f471000f300 con 0x7f47140f5410 2023-12-06T03:59:35.208 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:35.209 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:35.210 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:35.208+0000 7f471c3c9700 1 -- 172.21.15.152:0/1582828095 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f470002ff60 msgr2=0x7f4700032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:35.210 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:35.208+0000 7f471c3c9700 1 --2- 172.21.15.152:0/1582828095 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f470002ff60 0x7f4700032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f47040096a0 tx=0x7f4704006b40).stop 2023-12-06T03:59:35.210 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:35.208+0000 7f471c3c9700 1 -- 172.21.15.152:0/1582828095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47140f5410 msgr2=0x7f4714109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:35.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:35.208+0000 7f471c3c9700 1 --2- 172.21.15.152:0/1582828095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47140f5410 0x7f4714109450 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f4710014000 tx=0x7f471000b040).stop 2023-12-06T03:59:35.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:35.208+0000 7f471c3c9700 1 -- 172.21.15.152:0/1582828095 shutdown_connections 2023-12-06T03:59:35.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:35.208+0000 7f471c3c9700 1 --2- 172.21.15.152:0/1582828095 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f470002ff60 0x7f4700032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:35.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:35.208+0000 7f471c3c9700 1 --2- 172.21.15.152:0/1582828095 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f47140f5410 0x7f4714109450 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:35.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:35.208+0000 7f471c3c9700 1 -- 172.21.15.152:0/1582828095 >> 172.21.15.152:0/1582828095 conn(0x7f47140f0b70 msgr2=0x7f47140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:35.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:35.208+0000 7f471c3c9700 1 -- 172.21.15.152:0/1582828095 shutdown_connections 2023-12-06T03:59:35.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:35.208+0000 7f471c3c9700 1 -- 172.21.15.152:0/1582828095 wait complete. 2023-12-06T03:59:35.211 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:36.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:35 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:34.447607+0000 mgr.smithi132.gokxja (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:36.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:35 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:35.208211+0000 mon.smithi132 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.152:0/1582828095' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:36.589 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:36.589 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:37.705 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.701+0000 7fe133922700 1 -- 172.21.15.152:0/685098610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe12c0f6510 msgr2=0x7fe12c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:37.705 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.701+0000 7fe133922700 1 --2- 172.21.15.152:0/685098610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe12c0f6510 0x7fe12c0f6930 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fe120009fb0 tx=0x7fe120009b30).stop 2023-12-06T03:59:37.705 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.701+0000 7fe133922700 1 -- 172.21.15.152:0/685098610 shutdown_connections 2023-12-06T03:59:37.705 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.701+0000 7fe133922700 1 --2- 172.21.15.152:0/685098610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe12c0f6510 0x7fe12c0f6930 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:37.705 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.701+0000 7fe133922700 1 -- 172.21.15.152:0/685098610 >> 172.21.15.152:0/685098610 conn(0x7fe12c0f1c70 msgr2=0x7fe12c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:37.705 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.701+0000 7fe133922700 1 -- 172.21.15.152:0/685098610 shutdown_connections 2023-12-06T03:59:37.705 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.701+0000 7fe133922700 1 -- 172.21.15.152:0/685098610 wait complete. 2023-12-06T03:59:37.705 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.702+0000 7fe133922700 1 Processor -- start 2023-12-06T03:59:37.705 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.702+0000 7fe133922700 1 -- start start 2023-12-06T03:59:37.705 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.702+0000 7fe133922700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe12c0f6510 0x7fe12c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:37.705 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.702+0000 7fe133922700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe12c069e20 con 0x7fe12c0f6510 2023-12-06T03:59:37.705 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.702+0000 7fe1316be700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe12c0f6510 0x7fe12c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:37.706 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.702+0000 7fe1316be700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe12c0f6510 0x7fe12c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:36588/0 (socket says 172.21.15.152:36588) 2023-12-06T03:59:37.706 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.702+0000 7fe1316be700 1 -- 172.21.15.152:0/882159402 learned_addr learned my addr 172.21.15.152:0/882159402 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:37.706 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.703+0000 7fe1316be700 1 -- 172.21.15.152:0/882159402 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe120005040 con 0x7fe12c0f6510 2023-12-06T03:59:37.706 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.703+0000 7fe1316be700 1 --2- 172.21.15.152:0/882159402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe12c0f6510 0x7fe12c06b9a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fe120013510 tx=0x7fe12000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:37.706 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.703+0000 7fe11dffb700 1 -- 172.21.15.152:0/882159402 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe120009320 con 0x7fe12c0f6510 2023-12-06T03:59:37.706 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.703+0000 7fe133922700 1 -- 172.21.15.152:0/882159402 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe12c069fc0 con 0x7fe12c0f6510 2023-12-06T03:59:37.706 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.703+0000 7fe11dffb700 1 -- 172.21.15.152:0/882159402 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe120013f00 con 0x7fe12c0f6510 2023-12-06T03:59:37.706 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.703+0000 7fe11dffb700 1 -- 172.21.15.152:0/882159402 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe120014d90 con 0x7fe12c0f6510 2023-12-06T03:59:37.706 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.703+0000 7fe133922700 1 -- 172.21.15.152:0/882159402 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe12c06a450 con 0x7fe12c0f6510 2023-12-06T03:59:37.707 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.704+0000 7fe11dffb700 1 -- 172.21.15.152:0/882159402 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1200095f0 con 0x7fe12c0f6510 2023-12-06T03:59:37.707 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.704+0000 7fe133922700 1 -- 172.21.15.152:0/882159402 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe12c059100 con 0x7fe12c0f6510 2023-12-06T03:59:37.708 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.704+0000 7fe11dffb700 1 --2- 172.21.15.152:0/882159402 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe11802ff60 0x7fe118032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:37.708 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.704+0000 7fe11dffb700 1 -- 172.21.15.152:0/882159402 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe120041000 con 0x7fe12c0f6510 2023-12-06T03:59:37.708 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.705+0000 7fe130ebd700 1 --2- 172.21.15.152:0/882159402 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe11802ff60 0x7fe118032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:37.708 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.705+0000 7fe130ebd700 1 --2- 172.21.15.152:0/882159402 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe11802ff60 0x7fe118032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fe1280096a0 tx=0x7fe128006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:37.711 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.709+0000 7fe11dffb700 1 -- 172.21.15.152:0/882159402 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe120014420 con 0x7fe12c0f6510 2023-12-06T03:59:37.920 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.918+0000 7fe133922700 1 -- 172.21.15.152:0/882159402 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe12c0fbf60 con 0x7fe12c0f6510 2023-12-06T03:59:37.920 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.918+0000 7fe11dffb700 1 -- 172.21.15.152:0/882159402 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe12001d7f0 con 0x7fe12c0f6510 2023-12-06T03:59:37.920 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:37.920 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:37.922 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.920+0000 7fe133922700 1 -- 172.21.15.152:0/882159402 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe11802ff60 msgr2=0x7fe118032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:37.922 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.920+0000 7fe133922700 1 --2- 172.21.15.152:0/882159402 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe11802ff60 0x7fe118032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fe1280096a0 tx=0x7fe128006b40).stop 2023-12-06T03:59:37.922 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.920+0000 7fe133922700 1 -- 172.21.15.152:0/882159402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe12c0f6510 msgr2=0x7fe12c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:37.922 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.920+0000 7fe133922700 1 --2- 172.21.15.152:0/882159402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe12c0f6510 0x7fe12c06b9a0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fe120013510 tx=0x7fe12000b040).stop 2023-12-06T03:59:37.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.920+0000 7fe133922700 1 -- 172.21.15.152:0/882159402 shutdown_connections 2023-12-06T03:59:37.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.920+0000 7fe133922700 1 --2- 172.21.15.152:0/882159402 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe11802ff60 0x7fe118032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:37.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.920+0000 7fe133922700 1 --2- 172.21.15.152:0/882159402 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe12c0f6510 0x7fe12c06b9a0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:37.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.920+0000 7fe133922700 1 -- 172.21.15.152:0/882159402 >> 172.21.15.152:0/882159402 conn(0x7fe12c0f1c70 msgr2=0x7fe12c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:37.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.920+0000 7fe133922700 1 -- 172.21.15.152:0/882159402 shutdown_connections 2023-12-06T03:59:37.923 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:37.920+0000 7fe133922700 1 -- 172.21.15.152:0/882159402 wait complete. 2023-12-06T03:59:37.924 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:38.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:36.448007+0000 mgr.smithi132.gokxja (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:39.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:38 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:37.919954+0000 mon.smithi132 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.152:0/882159402' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:39.323 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:39.323 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:40.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:39 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:38.448372+0000 mgr.smithi132.gokxja (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:40.430 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.428+0000 7f2c75ec7700 1 -- 172.21.15.152:0/2512472516 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c700f5410 msgr2=0x7f2c700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:40.431 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.428+0000 7f2c75ec7700 1 --2- 172.21.15.152:0/2512472516 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c700f5410 0x7f2c700f5830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f2c60009fb0 tx=0x7f2c60009b30).stop 2023-12-06T03:59:40.431 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.429+0000 7f2c75ec7700 1 -- 172.21.15.152:0/2512472516 shutdown_connections 2023-12-06T03:59:40.431 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.429+0000 7f2c75ec7700 1 --2- 172.21.15.152:0/2512472516 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c700f5410 0x7f2c700f5830 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:40.431 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.429+0000 7f2c75ec7700 1 -- 172.21.15.152:0/2512472516 >> 172.21.15.152:0/2512472516 conn(0x7f2c700f0b70 msgr2=0x7f2c700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:40.431 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.429+0000 7f2c75ec7700 1 -- 172.21.15.152:0/2512472516 shutdown_connections 2023-12-06T03:59:40.431 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.429+0000 7f2c75ec7700 1 -- 172.21.15.152:0/2512472516 wait complete. 2023-12-06T03:59:40.433 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.429+0000 7f2c75ec7700 1 Processor -- start 2023-12-06T03:59:40.434 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.429+0000 7f2c75ec7700 1 -- start start 2023-12-06T03:59:40.434 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.429+0000 7f2c75ec7700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c700f5410 0x7f2c70109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:40.434 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.430+0000 7f2c75ec7700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c70060220 con 0x7f2c700f5410 2023-12-06T03:59:40.434 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.430+0000 7f2c6f7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c700f5410 0x7f2c70109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:40.434 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.430+0000 7f2c6f7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c700f5410 0x7f2c70109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:59430/0 (socket says 172.21.15.152:59430) 2023-12-06T03:59:40.434 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.430+0000 7f2c6f7fe700 1 -- 172.21.15.152:0/960215388 learned_addr learned my addr 172.21.15.152:0/960215388 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:40.434 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.430+0000 7f2c6f7fe700 1 -- 172.21.15.152:0/960215388 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c60005040 con 0x7f2c700f5410 2023-12-06T03:59:40.434 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.430+0000 7f2c6f7fe700 1 --2- 172.21.15.152:0/960215388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c700f5410 0x7f2c70109420 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f2c60013510 tx=0x7f2c6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:40.434 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.430+0000 7f2c6cff9700 1 -- 172.21.15.152:0/960215388 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c60009320 con 0x7f2c700f5410 2023-12-06T03:59:40.435 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.430+0000 7f2c75ec7700 1 -- 172.21.15.152:0/960215388 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c70060420 con 0x7f2c700f5410 2023-12-06T03:59:40.435 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.430+0000 7f2c6cff9700 1 -- 172.21.15.152:0/960215388 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2c60013f00 con 0x7f2c700f5410 2023-12-06T03:59:40.435 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.430+0000 7f2c6cff9700 1 -- 172.21.15.152:0/960215388 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c60014d90 con 0x7f2c700f5410 2023-12-06T03:59:40.435 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.430+0000 7f2c75ec7700 1 -- 172.21.15.152:0/960215388 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c70109ad0 con 0x7f2c700f5410 2023-12-06T03:59:40.435 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.431+0000 7f2c6cff9700 1 -- 172.21.15.152:0/960215388 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2c600095f0 con 0x7f2c700f5410 2023-12-06T03:59:40.435 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.431+0000 7f2c75ec7700 1 -- 172.21.15.152:0/960215388 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c70059100 con 0x7f2c700f5410 2023-12-06T03:59:40.435 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.431+0000 7f2c6cff9700 1 --2- 172.21.15.152:0/960215388 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2c5c02ff60 0x7f2c5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:40.435 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.431+0000 7f2c6cff9700 1 -- 172.21.15.152:0/960215388 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2c600410c0 con 0x7f2c700f5410 2023-12-06T03:59:40.435 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.431+0000 7f2c67fff700 1 --2- 172.21.15.152:0/960215388 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2c5c02ff60 0x7f2c5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:40.435 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.432+0000 7f2c67fff700 1 --2- 172.21.15.152:0/960215388 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2c5c02ff60 0x7f2c5c032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f2c580096a0 tx=0x7f2c58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:40.438 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.436+0000 7f2c6cff9700 1 -- 172.21.15.152:0/960215388 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2c60014420 con 0x7f2c700f5410 2023-12-06T03:59:40.645 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.642+0000 7f2c75ec7700 1 -- 172.21.15.152:0/960215388 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c70045ad0 con 0x7f2c700f5410 2023-12-06T03:59:40.645 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.643+0000 7f2c6cff9700 1 -- 172.21.15.152:0/960215388 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2c60010230 con 0x7f2c700f5410 2023-12-06T03:59:40.646 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:40.646 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:40.647 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.645+0000 7f2c75ec7700 1 -- 172.21.15.152:0/960215388 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2c5c02ff60 msgr2=0x7f2c5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:40.647 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.645+0000 7f2c75ec7700 1 --2- 172.21.15.152:0/960215388 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2c5c02ff60 0x7f2c5c032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f2c580096a0 tx=0x7f2c58006b40).stop 2023-12-06T03:59:40.647 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.645+0000 7f2c75ec7700 1 -- 172.21.15.152:0/960215388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c700f5410 msgr2=0x7f2c70109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:40.647 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.645+0000 7f2c75ec7700 1 --2- 172.21.15.152:0/960215388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c700f5410 0x7f2c70109420 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f2c60013510 tx=0x7f2c6000b040).stop 2023-12-06T03:59:40.647 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.645+0000 7f2c75ec7700 1 -- 172.21.15.152:0/960215388 shutdown_connections 2023-12-06T03:59:40.647 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.645+0000 7f2c75ec7700 1 --2- 172.21.15.152:0/960215388 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f2c5c02ff60 0x7f2c5c032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:40.647 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.645+0000 7f2c75ec7700 1 --2- 172.21.15.152:0/960215388 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2c700f5410 0x7f2c70109420 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:40.648 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.645+0000 7f2c75ec7700 1 -- 172.21.15.152:0/960215388 >> 172.21.15.152:0/960215388 conn(0x7f2c700f0b70 msgr2=0x7f2c700f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:40.648 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.645+0000 7f2c75ec7700 1 -- 172.21.15.152:0/960215388 shutdown_connections 2023-12-06T03:59:40.648 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:40.645+0000 7f2c75ec7700 1 -- 172.21.15.152:0/960215388 wait complete. 2023-12-06T03:59:40.648 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:42.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:42.025 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:42.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:41 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:40.448737+0000 mgr.smithi132.gokxja (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:42.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:41 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:40.644726+0000 mon.smithi132 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.152:0/960215388' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:43.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.219+0000 7f23dcba2700 1 -- 172.21.15.152:0/3187792156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23d80f51a0 msgr2=0x7f23d80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:43.223 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.219+0000 7f23dcba2700 1 --2- 172.21.15.152:0/3187792156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23d80f51a0 0x7f23d80f55c0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f23c8009fb0 tx=0x7f23c8009b30).stop 2023-12-06T03:59:43.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.219+0000 7f23dcba2700 1 -- 172.21.15.152:0/3187792156 shutdown_connections 2023-12-06T03:59:43.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.219+0000 7f23dcba2700 1 --2- 172.21.15.152:0/3187792156 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23d80f51a0 0x7f23d80f55c0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:43.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.219+0000 7f23dcba2700 1 -- 172.21.15.152:0/3187792156 >> 172.21.15.152:0/3187792156 conn(0x7f23d80f09a0 msgr2=0x7f23d80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:43.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.219+0000 7f23dcba2700 1 -- 172.21.15.152:0/3187792156 shutdown_connections 2023-12-06T03:59:43.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.219+0000 7f23dcba2700 1 -- 172.21.15.152:0/3187792156 wait complete. 2023-12-06T03:59:43.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.219+0000 7f23dcba2700 1 Processor -- start 2023-12-06T03:59:43.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.219+0000 7f23dcba2700 1 -- start start 2023-12-06T03:59:43.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.219+0000 7f23dcba2700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23d80f51a0 0x7f23d81064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:43.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.220+0000 7f23dcba2700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23d81069b0 con 0x7f23d80f51a0 2023-12-06T03:59:43.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.220+0000 7f23d77fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23d80f51a0 0x7f23d81064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:43.224 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.220+0000 7f23d77fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23d80f51a0 0x7f23d81064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:59442/0 (socket says 172.21.15.152:59442) 2023-12-06T03:59:43.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.220+0000 7f23d77fe700 1 -- 172.21.15.152:0/2368155505 learned_addr learned my addr 172.21.15.152:0/2368155505 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:43.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.220+0000 7f23d77fe700 1 -- 172.21.15.152:0/2368155505 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23c8005040 con 0x7f23d80f51a0 2023-12-06T03:59:43.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.220+0000 7f23d77fe700 1 --2- 172.21.15.152:0/2368155505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23d80f51a0 0x7f23d81064c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f23c8013510 tx=0x7f23c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:43.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.220+0000 7f23bffff700 1 -- 172.21.15.152:0/2368155505 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23c8009320 con 0x7f23d80f51a0 2023-12-06T03:59:43.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.220+0000 7f23bffff700 1 -- 172.21.15.152:0/2368155505 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23c8013f00 con 0x7f23d80f51a0 2023-12-06T03:59:43.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.220+0000 7f23dcba2700 1 -- 172.21.15.152:0/2368155505 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23d8106bb0 con 0x7f23d80f51a0 2023-12-06T03:59:43.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.220+0000 7f23bffff700 1 -- 172.21.15.152:0/2368155505 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f23c8014dc0 con 0x7f23d80f51a0 2023-12-06T03:59:43.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.220+0000 7f23dcba2700 1 -- 172.21.15.152:0/2368155505 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23d8107040 con 0x7f23d80f51a0 2023-12-06T03:59:43.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.221+0000 7f23bffff700 1 -- 172.21.15.152:0/2368155505 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f23c8014420 con 0x7f23d80f51a0 2023-12-06T03:59:43.225 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.221+0000 7f23dcba2700 1 -- 172.21.15.152:0/2368155505 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23d8045ad0 con 0x7f23d80f51a0 2023-12-06T03:59:43.226 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.221+0000 7f23bffff700 1 --2- 172.21.15.152:0/2368155505 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f23c002ff60 0x7f23c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:43.226 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.221+0000 7f23bffff700 1 -- 172.21.15.152:0/2368155505 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f23c80410f0 con 0x7f23d80f51a0 2023-12-06T03:59:43.226 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.222+0000 7f23d6ffd700 1 --2- 172.21.15.152:0/2368155505 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f23c002ff60 0x7f23c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:43.226 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.222+0000 7f23d6ffd700 1 --2- 172.21.15.152:0/2368155505 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f23c002ff60 0x7f23c0032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f23cc0096a0 tx=0x7f23cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:43.228 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.226+0000 7f23bffff700 1 -- 172.21.15.152:0/2368155505 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23c80095f0 con 0x7f23d80f51a0 2023-12-06T03:59:43.433 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.432+0000 7f23dcba2700 1 -- 172.21.15.152:0/2368155505 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23d8107ef0 con 0x7f23d80f51a0 2023-12-06T03:59:43.434 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.432+0000 7f23bffff700 1 -- 172.21.15.152:0/2368155505 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f23c801cb10 con 0x7f23d80f51a0 2023-12-06T03:59:43.434 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:43.434 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:43.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.434+0000 7f23dcba2700 1 -- 172.21.15.152:0/2368155505 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f23c002ff60 msgr2=0x7f23c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:43.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.434+0000 7f23dcba2700 1 --2- 172.21.15.152:0/2368155505 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f23c002ff60 0x7f23c0032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f23cc0096a0 tx=0x7f23cc006b40).stop 2023-12-06T03:59:43.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.434+0000 7f23dcba2700 1 -- 172.21.15.152:0/2368155505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23d80f51a0 msgr2=0x7f23d81064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:43.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.434+0000 7f23dcba2700 1 --2- 172.21.15.152:0/2368155505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23d80f51a0 0x7f23d81064c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f23c8013510 tx=0x7f23c800b040).stop 2023-12-06T03:59:43.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.434+0000 7f23dcba2700 1 -- 172.21.15.152:0/2368155505 shutdown_connections 2023-12-06T03:59:43.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.434+0000 7f23dcba2700 1 --2- 172.21.15.152:0/2368155505 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f23c002ff60 0x7f23c0032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:43.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.434+0000 7f23dcba2700 1 --2- 172.21.15.152:0/2368155505 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f23d80f51a0 0x7f23d81064c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:43.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.434+0000 7f23dcba2700 1 -- 172.21.15.152:0/2368155505 >> 172.21.15.152:0/2368155505 conn(0x7f23d80f09a0 msgr2=0x7f23d80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:43.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.434+0000 7f23dcba2700 1 -- 172.21.15.152:0/2368155505 shutdown_connections 2023-12-06T03:59:43.436 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:43.434+0000 7f23dcba2700 1 -- 172.21.15.152:0/2368155505 wait complete. 2023-12-06T03:59:43.437 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:44.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:42.449084+0000 mgr.smithi132.gokxja (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:44.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:43.433827+0000 mon.smithi132 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.152:0/2368155505' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:44.805 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:44.805 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:45.904 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.899+0000 7f71dfbca700 1 -- 172.21.15.152:0/2672348673 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71d8060280 msgr2=0x7f71d80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:45.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.899+0000 7f71dfbca700 1 --2- 172.21.15.152:0/2672348673 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71d8060280 0x7f71d80f90f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f71c8009fb0 tx=0x7f71c8009b30).stop 2023-12-06T03:59:45.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.899+0000 7f71dfbca700 1 -- 172.21.15.152:0/2672348673 shutdown_connections 2023-12-06T03:59:45.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.899+0000 7f71dfbca700 1 --2- 172.21.15.152:0/2672348673 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71d8060280 0x7f71d80f90f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:45.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.899+0000 7f71dfbca700 1 -- 172.21.15.152:0/2672348673 >> 172.21.15.152:0/2672348673 conn(0x7f71d80f0b70 msgr2=0x7f71d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:45.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.899+0000 7f71dfbca700 1 -- 172.21.15.152:0/2672348673 shutdown_connections 2023-12-06T03:59:45.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.899+0000 7f71dfbca700 1 -- 172.21.15.152:0/2672348673 wait complete. 2023-12-06T03:59:45.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.900+0000 7f71dfbca700 1 Processor -- start 2023-12-06T03:59:45.906 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.900+0000 7f71dfbca700 1 -- start start 2023-12-06T03:59:45.906 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.900+0000 7f71dfbca700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71d8060280 0x7f71d81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:45.906 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.900+0000 7f71dfbca700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d8106bd0 con 0x7f71d8060280 2023-12-06T03:59:45.907 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.900+0000 7f71dd966700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71d8060280 0x7f71d81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:45.907 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.900+0000 7f71dd966700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71d8060280 0x7f71d81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:59458/0 (socket says 172.21.15.152:59458) 2023-12-06T03:59:45.907 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.900+0000 7f71dd966700 1 -- 172.21.15.152:0/3271865865 learned_addr learned my addr 172.21.15.152:0/3271865865 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:45.907 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.900+0000 7f71dd966700 1 -- 172.21.15.152:0/3271865865 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c8005040 con 0x7f71d8060280 2023-12-06T03:59:45.907 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.900+0000 7f71dd966700 1 --2- 172.21.15.152:0/3271865865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71d8060280 0x7f71d81066e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f71c8009830 tx=0x7f71c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:45.907 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.901+0000 7f71ce7fc700 1 -- 172.21.15.152:0/3271865865 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71c8009320 con 0x7f71d8060280 2023-12-06T03:59:45.907 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.901+0000 7f71ce7fc700 1 -- 172.21.15.152:0/3271865865 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71c80136c0 con 0x7f71d8060280 2023-12-06T03:59:45.908 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.901+0000 7f71dfbca700 1 -- 172.21.15.152:0/3271865865 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d8106dd0 con 0x7f71d8060280 2023-12-06T03:59:45.908 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.901+0000 7f71ce7fc700 1 -- 172.21.15.152:0/3271865865 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71c801ee90 con 0x7f71d8060280 2023-12-06T03:59:45.908 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.901+0000 7f71dfbca700 1 -- 172.21.15.152:0/3271865865 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d8107260 con 0x7f71d8060280 2023-12-06T03:59:45.908 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.901+0000 7f71dfbca700 1 -- 172.21.15.152:0/3271865865 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d8045ad0 con 0x7f71d8060280 2023-12-06T03:59:45.908 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.901+0000 7f71ce7fc700 1 -- 172.21.15.152:0/3271865865 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71c801e420 con 0x7f71d8060280 2023-12-06T03:59:45.908 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.902+0000 7f71ce7fc700 1 --2- 172.21.15.152:0/3271865865 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f71c402ff10 0x7f71c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:45.909 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.902+0000 7f71ce7fc700 1 -- 172.21.15.152:0/3271865865 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f71c8040ff0 con 0x7f71d8060280 2023-12-06T03:59:45.909 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.902+0000 7f71dd165700 1 --2- 172.21.15.152:0/3271865865 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f71c402ff10 0x7f71c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:45.909 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.902+0000 7f71dd165700 1 --2- 172.21.15.152:0/3271865865 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f71c402ff10 0x7f71c40323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f71d40096a0 tx=0x7f71d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:45.910 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:45.906+0000 7f71ce7fc700 1 -- 172.21.15.152:0/3271865865 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f71c806d2b0 con 0x7f71d8060280 2023-12-06T03:59:46.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:45 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:44.449514+0000 mgr.smithi132.gokxja (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:46.115 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:46.113+0000 7f71dfbca700 1 -- 172.21.15.152:0/3271865865 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71d8108110 con 0x7f71d8060280 2023-12-06T03:59:46.116 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:46.115+0000 7f71ce7fc700 1 -- 172.21.15.152:0/3271865865 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f71c802c2d0 con 0x7f71d8060280 2023-12-06T03:59:46.116 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:46.116 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:46.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:46.116+0000 7f71dfbca700 1 -- 172.21.15.152:0/3271865865 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f71c402ff10 msgr2=0x7f71c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:46.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:46.116+0000 7f71dfbca700 1 --2- 172.21.15.152:0/3271865865 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f71c402ff10 0x7f71c40323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f71d40096a0 tx=0x7f71d4006b40).stop 2023-12-06T03:59:46.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:46.116+0000 7f71dfbca700 1 -- 172.21.15.152:0/3271865865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71d8060280 msgr2=0x7f71d81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:46.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:46.116+0000 7f71dfbca700 1 --2- 172.21.15.152:0/3271865865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71d8060280 0x7f71d81066e0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f71c8009830 tx=0x7f71c800b040).stop 2023-12-06T03:59:46.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:46.116+0000 7f71dfbca700 1 -- 172.21.15.152:0/3271865865 shutdown_connections 2023-12-06T03:59:46.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:46.116+0000 7f71dfbca700 1 --2- 172.21.15.152:0/3271865865 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f71c402ff10 0x7f71c40323d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:46.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:46.116+0000 7f71dfbca700 1 --2- 172.21.15.152:0/3271865865 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f71d8060280 0x7f71d81066e0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:46.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:46.116+0000 7f71dfbca700 1 -- 172.21.15.152:0/3271865865 >> 172.21.15.152:0/3271865865 conn(0x7f71d80f0b70 msgr2=0x7f71d80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:46.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:46.116+0000 7f71dfbca700 1 -- 172.21.15.152:0/3271865865 shutdown_connections 2023-12-06T03:59:46.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:46.116+0000 7f71dfbca700 1 -- 172.21.15.152:0/3271865865 wait complete. 2023-12-06T03:59:46.119 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:47.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:46 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:46.115598+0000 mon.smithi132 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.152:0/3271865865' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:47.480 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:47.480 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:48.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:47 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:46.449934+0000 mgr.smithi132.gokxja (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:48.633 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.631+0000 7fad2911c700 1 -- 172.21.15.152:0/957376996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad240f6510 msgr2=0x7fad240f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:48.633 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.631+0000 7fad2911c700 1 --2- 172.21.15.152:0/957376996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad240f6510 0x7fad240f6930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7fad14009fb0 tx=0x7fad14009b30).stop 2023-12-06T03:59:48.633 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.631+0000 7fad2911c700 1 -- 172.21.15.152:0/957376996 shutdown_connections 2023-12-06T03:59:48.634 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.631+0000 7fad2911c700 1 --2- 172.21.15.152:0/957376996 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad240f6510 0x7fad240f6930 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:48.634 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.631+0000 7fad2911c700 1 -- 172.21.15.152:0/957376996 >> 172.21.15.152:0/957376996 conn(0x7fad240f1c70 msgr2=0x7fad240f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:48.634 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.631+0000 7fad2911c700 1 -- 172.21.15.152:0/957376996 shutdown_connections 2023-12-06T03:59:48.634 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.631+0000 7fad2911c700 1 -- 172.21.15.152:0/957376996 wait complete. 2023-12-06T03:59:48.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.632+0000 7fad2911c700 1 Processor -- start 2023-12-06T03:59:48.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.632+0000 7fad2911c700 1 -- start start 2023-12-06T03:59:48.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.632+0000 7fad2911c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad240f6510 0x7fad24106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:48.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.632+0000 7fad2911c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad24106c20 con 0x7fad240f6510 2023-12-06T03:59:48.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.632+0000 7fad22d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad240f6510 0x7fad24106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:48.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.632+0000 7fad22d9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad240f6510 0x7fad24106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:59466/0 (socket says 172.21.15.152:59466) 2023-12-06T03:59:48.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.632+0000 7fad22d9d700 1 -- 172.21.15.152:0/2080138174 learned_addr learned my addr 172.21.15.152:0/2080138174 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:48.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.633+0000 7fad22d9d700 1 -- 172.21.15.152:0/2080138174 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad14005040 con 0x7fad240f6510 2023-12-06T03:59:48.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.633+0000 7fad22d9d700 1 --2- 172.21.15.152:0/2080138174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad240f6510 0x7fad24106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fad14013510 tx=0x7fad1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:48.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.633+0000 7fad1b7fe700 1 -- 172.21.15.152:0/2080138174 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad14009320 con 0x7fad240f6510 2023-12-06T03:59:48.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.633+0000 7fad2911c700 1 -- 172.21.15.152:0/2080138174 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad24106e20 con 0x7fad240f6510 2023-12-06T03:59:48.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.633+0000 7fad1b7fe700 1 -- 172.21.15.152:0/2080138174 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fad14013f00 con 0x7fad240f6510 2023-12-06T03:59:48.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.633+0000 7fad1b7fe700 1 -- 172.21.15.152:0/2080138174 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad14014d90 con 0x7fad240f6510 2023-12-06T03:59:48.637 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.633+0000 7fad2911c700 1 -- 172.21.15.152:0/2080138174 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad241072b0 con 0x7fad240f6510 2023-12-06T03:59:48.637 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.634+0000 7fad1b7fe700 1 -- 172.21.15.152:0/2080138174 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fad140095f0 con 0x7fad240f6510 2023-12-06T03:59:48.637 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.634+0000 7fad2911c700 1 -- 172.21.15.152:0/2080138174 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad24045ad0 con 0x7fad240f6510 2023-12-06T03:59:48.637 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.634+0000 7fad1b7fe700 1 --2- 172.21.15.152:0/2080138174 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fad1002ff60 0x7fad10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:48.637 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.634+0000 7fad1b7fe700 1 -- 172.21.15.152:0/2080138174 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fad140410c0 con 0x7fad240f6510 2023-12-06T03:59:48.637 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.634+0000 7fad2259c700 1 --2- 172.21.15.152:0/2080138174 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fad1002ff60 0x7fad10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:48.637 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.635+0000 7fad2259c700 1 --2- 172.21.15.152:0/2080138174 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fad1002ff60 0x7fad10032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fad0c0096a0 tx=0x7fad0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:48.640 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.639+0000 7fad1b7fe700 1 -- 172.21.15.152:0/2080138174 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fad14014420 con 0x7fad240f6510 2023-12-06T03:59:48.849 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.847+0000 7fad2911c700 1 -- 172.21.15.152:0/2080138174 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad240fbf60 con 0x7fad240f6510 2023-12-06T03:59:48.849 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.848+0000 7fad1b7fe700 1 -- 172.21.15.152:0/2080138174 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fad14010230 con 0x7fad240f6510 2023-12-06T03:59:48.850 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:48.850 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:48.851 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.849+0000 7fad2911c700 1 -- 172.21.15.152:0/2080138174 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fad1002ff60 msgr2=0x7fad10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:48.851 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.849+0000 7fad2911c700 1 --2- 172.21.15.152:0/2080138174 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fad1002ff60 0x7fad10032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fad0c0096a0 tx=0x7fad0c006b40).stop 2023-12-06T03:59:48.851 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.849+0000 7fad2911c700 1 -- 172.21.15.152:0/2080138174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad240f6510 msgr2=0x7fad24106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:48.851 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.849+0000 7fad2911c700 1 --2- 172.21.15.152:0/2080138174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad240f6510 0x7fad24106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7fad14013510 tx=0x7fad1400b040).stop 2023-12-06T03:59:48.851 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.849+0000 7fad2911c700 1 -- 172.21.15.152:0/2080138174 shutdown_connections 2023-12-06T03:59:48.851 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.849+0000 7fad2911c700 1 --2- 172.21.15.152:0/2080138174 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fad1002ff60 0x7fad10032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:48.852 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.849+0000 7fad2911c700 1 --2- 172.21.15.152:0/2080138174 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fad240f6510 0x7fad24106730 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:48.852 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.849+0000 7fad2911c700 1 -- 172.21.15.152:0/2080138174 >> 172.21.15.152:0/2080138174 conn(0x7fad240f1c70 msgr2=0x7fad240f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:48.852 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.849+0000 7fad2911c700 1 -- 172.21.15.152:0/2080138174 shutdown_connections 2023-12-06T03:59:48.852 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:48.849+0000 7fad2911c700 1 -- 172.21.15.152:0/2080138174 wait complete. 2023-12-06T03:59:48.852 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:50.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:49 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:48.450324+0000 mgr.smithi132.gokxja (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:50.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:49 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:48.849088+0000 mon.smithi132 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.152:0/2080138174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:50.226 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:50.227 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:51.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.345+0000 7fcc60ef3700 1 -- 172.21.15.152:0/1730945750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc5c060280 msgr2=0x7fcc5c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:51.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.345+0000 7fcc60ef3700 1 --2- 172.21.15.152:0/1730945750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc5c060280 0x7fcc5c0f90f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7fcc50009fb0 tx=0x7fcc50009b30).stop 2023-12-06T03:59:51.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.345+0000 7fcc60ef3700 1 -- 172.21.15.152:0/1730945750 shutdown_connections 2023-12-06T03:59:51.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.345+0000 7fcc60ef3700 1 --2- 172.21.15.152:0/1730945750 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc5c060280 0x7fcc5c0f90f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:51.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.345+0000 7fcc60ef3700 1 -- 172.21.15.152:0/1730945750 >> 172.21.15.152:0/1730945750 conn(0x7fcc5c0f0b70 msgr2=0x7fcc5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:51.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.345+0000 7fcc60ef3700 1 -- 172.21.15.152:0/1730945750 shutdown_connections 2023-12-06T03:59:51.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.345+0000 7fcc60ef3700 1 -- 172.21.15.152:0/1730945750 wait complete. 2023-12-06T03:59:51.348 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.345+0000 7fcc60ef3700 1 Processor -- start 2023-12-06T03:59:51.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.345+0000 7fcc60ef3700 1 -- start start 2023-12-06T03:59:51.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.346+0000 7fcc60ef3700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc5c060280 0x7fcc5c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:51.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.346+0000 7fcc60ef3700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc5c106bd0 con 0x7fcc5c060280 2023-12-06T03:59:51.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.346+0000 7fcc5a59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc5c060280 0x7fcc5c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:51.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.346+0000 7fcc5a59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc5c060280 0x7fcc5c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:49452/0 (socket says 172.21.15.152:49452) 2023-12-06T03:59:51.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.346+0000 7fcc5a59c700 1 -- 172.21.15.152:0/3876354184 learned_addr learned my addr 172.21.15.152:0/3876354184 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:51.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.346+0000 7fcc5a59c700 1 -- 172.21.15.152:0/3876354184 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc50005040 con 0x7fcc5c060280 2023-12-06T03:59:51.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.346+0000 7fcc5a59c700 1 --2- 172.21.15.152:0/3876354184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc5c060280 0x7fcc5c1066e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fcc50013510 tx=0x7fcc5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:51.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.346+0000 7fcc4affd700 1 -- 172.21.15.152:0/3876354184 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc50009320 con 0x7fcc5c060280 2023-12-06T03:59:51.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.346+0000 7fcc60ef3700 1 -- 172.21.15.152:0/3876354184 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc5c106dd0 con 0x7fcc5c060280 2023-12-06T03:59:51.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.346+0000 7fcc4affd700 1 -- 172.21.15.152:0/3876354184 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc50013f00 con 0x7fcc5c060280 2023-12-06T03:59:51.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.346+0000 7fcc4affd700 1 -- 172.21.15.152:0/3876354184 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc50014dc0 con 0x7fcc5c060280 2023-12-06T03:59:51.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.346+0000 7fcc60ef3700 1 -- 172.21.15.152:0/3876354184 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc5c107260 con 0x7fcc5c060280 2023-12-06T03:59:51.351 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.347+0000 7fcc4affd700 1 -- 172.21.15.152:0/3876354184 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc500095f0 con 0x7fcc5c060280 2023-12-06T03:59:51.351 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.347+0000 7fcc60ef3700 1 -- 172.21.15.152:0/3876354184 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc5c045ad0 con 0x7fcc5c060280 2023-12-06T03:59:51.351 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.347+0000 7fcc4affd700 1 --2- 172.21.15.152:0/3876354184 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcc4402ff60 0x7fcc44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:51.351 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.348+0000 7fcc4affd700 1 -- 172.21.15.152:0/3876354184 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc50041040 con 0x7fcc5c060280 2023-12-06T03:59:51.351 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.348+0000 7fcc59d9b700 1 --2- 172.21.15.152:0/3876354184 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcc4402ff60 0x7fcc44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:51.351 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.348+0000 7fcc59d9b700 1 --2- 172.21.15.152:0/3876354184 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcc4402ff60 0x7fcc44032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fcc4c0096a0 tx=0x7fcc4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:51.353 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.352+0000 7fcc4affd700 1 -- 172.21.15.152:0/3876354184 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc50014420 con 0x7fcc5c060280 2023-12-06T03:59:51.564 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.561+0000 7fcc60ef3700 1 -- 172.21.15.152:0/3876354184 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc5c108110 con 0x7fcc5c060280 2023-12-06T03:59:51.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.562+0000 7fcc4affd700 1 -- 172.21.15.152:0/3876354184 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcc50010230 con 0x7fcc5c060280 2023-12-06T03:59:51.565 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:51.565 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:51.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.563+0000 7fcc60ef3700 1 -- 172.21.15.152:0/3876354184 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcc4402ff60 msgr2=0x7fcc44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:51.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.563+0000 7fcc60ef3700 1 --2- 172.21.15.152:0/3876354184 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcc4402ff60 0x7fcc44032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fcc4c0096a0 tx=0x7fcc4c006b40).stop 2023-12-06T03:59:51.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.564+0000 7fcc60ef3700 1 -- 172.21.15.152:0/3876354184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc5c060280 msgr2=0x7fcc5c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:51.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.564+0000 7fcc60ef3700 1 --2- 172.21.15.152:0/3876354184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc5c060280 0x7fcc5c1066e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7fcc50013510 tx=0x7fcc5000b040).stop 2023-12-06T03:59:51.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.564+0000 7fcc60ef3700 1 -- 172.21.15.152:0/3876354184 shutdown_connections 2023-12-06T03:59:51.567 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.564+0000 7fcc60ef3700 1 --2- 172.21.15.152:0/3876354184 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcc4402ff60 0x7fcc44032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:51.567 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.564+0000 7fcc60ef3700 1 --2- 172.21.15.152:0/3876354184 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc5c060280 0x7fcc5c1066e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:51.567 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.564+0000 7fcc60ef3700 1 -- 172.21.15.152:0/3876354184 >> 172.21.15.152:0/3876354184 conn(0x7fcc5c0f0b70 msgr2=0x7fcc5c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:51.567 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.564+0000 7fcc60ef3700 1 -- 172.21.15.152:0/3876354184 shutdown_connections 2023-12-06T03:59:51.567 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:51.564+0000 7fcc60ef3700 1 -- 172.21.15.152:0/3876354184 wait complete. 2023-12-06T03:59:51.567 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:52.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:51 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:50.450815+0000 mgr.smithi132.gokxja (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:52.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:51 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:51.563486+0000 mon.smithi132 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.152:0/3876354184' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:52.951 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:52.951 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:54.024 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.018+0000 7fc3d38da700 1 -- 172.21.15.152:0/4092564173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3cc0f4b00 msgr2=0x7fc3cc0f4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:54.024 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.018+0000 7fc3d38da700 1 --2- 172.21.15.152:0/4092564173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3cc0f4b00 0x7fc3cc0f4f20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fc3bc009fb0 tx=0x7fc3bc009b30).stop 2023-12-06T03:59:54.024 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.018+0000 7fc3d38da700 1 -- 172.21.15.152:0/4092564173 shutdown_connections 2023-12-06T03:59:54.024 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.018+0000 7fc3d38da700 1 --2- 172.21.15.152:0/4092564173 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3cc0f4b00 0x7fc3cc0f4f20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:54.025 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.018+0000 7fc3d38da700 1 -- 172.21.15.152:0/4092564173 >> 172.21.15.152:0/4092564173 conn(0x7fc3cc0f0260 msgr2=0x7fc3cc0f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:54.026 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.018+0000 7fc3d38da700 1 -- 172.21.15.152:0/4092564173 shutdown_connections 2023-12-06T03:59:54.026 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.018+0000 7fc3d38da700 1 -- 172.21.15.152:0/4092564173 wait complete. 2023-12-06T03:59:54.026 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.019+0000 7fc3d38da700 1 Processor -- start 2023-12-06T03:59:54.026 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.019+0000 7fc3d38da700 1 -- start start 2023-12-06T03:59:54.026 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.019+0000 7fc3d38da700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3cc0f4b00 0x7fc3cc10b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:54.026 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.019+0000 7fc3d38da700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3cc107210 con 0x7fc3cc0f4b00 2023-12-06T03:59:54.027 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.019+0000 7fc3d1676700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3cc0f4b00 0x7fc3cc10b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:54.027 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.019+0000 7fc3d1676700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3cc0f4b00 0x7fc3cc10b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:49470/0 (socket says 172.21.15.152:49470) 2023-12-06T03:59:54.027 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.019+0000 7fc3d1676700 1 -- 172.21.15.152:0/1856365190 learned_addr learned my addr 172.21.15.152:0/1856365190 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:54.027 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.019+0000 7fc3d1676700 1 -- 172.21.15.152:0/1856365190 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3bc005040 con 0x7fc3cc0f4b00 2023-12-06T03:59:54.027 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.019+0000 7fc3d1676700 1 --2- 172.21.15.152:0/1856365190 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3cc0f4b00 0x7fc3cc10b820 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fc3bc013510 tx=0x7fc3bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:54.027 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.020+0000 7fc3c1ffb700 1 -- 172.21.15.152:0/1856365190 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3bc009320 con 0x7fc3cc0f4b00 2023-12-06T03:59:54.027 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.020+0000 7fc3c1ffb700 1 -- 172.21.15.152:0/1856365190 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3bc013f00 con 0x7fc3cc0f4b00 2023-12-06T03:59:54.028 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.020+0000 7fc3c1ffb700 1 -- 172.21.15.152:0/1856365190 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3bc014d90 con 0x7fc3cc0f4b00 2023-12-06T03:59:54.028 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.020+0000 7fc3d38da700 1 -- 172.21.15.152:0/1856365190 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3cc0f1870 con 0x7fc3cc0f4b00 2023-12-06T03:59:54.028 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.020+0000 7fc3d38da700 1 -- 172.21.15.152:0/1856365190 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3cc10bd10 con 0x7fc3cc0f4b00 2023-12-06T03:59:54.028 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.021+0000 7fc3c1ffb700 1 -- 172.21.15.152:0/1856365190 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3bc0095f0 con 0x7fc3cc0f4b00 2023-12-06T03:59:54.028 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.021+0000 7fc3d38da700 1 -- 172.21.15.152:0/1856365190 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3cc045ad0 con 0x7fc3cc0f4b00 2023-12-06T03:59:54.028 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.021+0000 7fc3c1ffb700 1 --2- 172.21.15.152:0/1856365190 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc3b802ffb0 0x7fc3b8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:54.028 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.021+0000 7fc3c1ffb700 1 -- 172.21.15.152:0/1856365190 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc3bc040ff0 con 0x7fc3cc0f4b00 2023-12-06T03:59:54.029 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.021+0000 7fc3d0e75700 1 --2- 172.21.15.152:0/1856365190 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc3b802ffb0 0x7fc3b8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:54.029 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.021+0000 7fc3d0e75700 1 --2- 172.21.15.152:0/1856365190 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc3b802ffb0 0x7fc3b8032470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc3c80096a0 tx=0x7fc3c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:54.030 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.026+0000 7fc3c1ffb700 1 -- 172.21.15.152:0/1856365190 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc3bc06d2b0 con 0x7fc3cc0f4b00 2023-12-06T03:59:54.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:53 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:52.451257+0000 mgr.smithi132.gokxja (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:54.232 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.230+0000 7fc3d38da700 1 -- 172.21.15.152:0/1856365190 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3cc027a00 con 0x7fc3cc0f4b00 2023-12-06T03:59:54.232 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.231+0000 7fc3c1ffb700 1 -- 172.21.15.152:0/1856365190 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc3bc014420 con 0x7fc3cc0f4b00 2023-12-06T03:59:54.232 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:54.233 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:54.234 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.232+0000 7fc3d38da700 1 -- 172.21.15.152:0/1856365190 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc3b802ffb0 msgr2=0x7fc3b8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:54.234 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.232+0000 7fc3d38da700 1 --2- 172.21.15.152:0/1856365190 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc3b802ffb0 0x7fc3b8032470 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fc3c80096a0 tx=0x7fc3c8006b40).stop 2023-12-06T03:59:54.234 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.232+0000 7fc3d38da700 1 -- 172.21.15.152:0/1856365190 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3cc0f4b00 msgr2=0x7fc3cc10b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:54.234 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.232+0000 7fc3d38da700 1 --2- 172.21.15.152:0/1856365190 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3cc0f4b00 0x7fc3cc10b820 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fc3bc013510 tx=0x7fc3bc00b040).stop 2023-12-06T03:59:54.234 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.232+0000 7fc3d38da700 1 -- 172.21.15.152:0/1856365190 shutdown_connections 2023-12-06T03:59:54.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.232+0000 7fc3d38da700 1 --2- 172.21.15.152:0/1856365190 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc3b802ffb0 0x7fc3b8032470 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:54.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.232+0000 7fc3d38da700 1 --2- 172.21.15.152:0/1856365190 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc3cc0f4b00 0x7fc3cc10b820 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:54.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.232+0000 7fc3d38da700 1 -- 172.21.15.152:0/1856365190 >> 172.21.15.152:0/1856365190 conn(0x7fc3cc0f0260 msgr2=0x7fc3cc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:54.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.232+0000 7fc3d38da700 1 -- 172.21.15.152:0/1856365190 shutdown_connections 2023-12-06T03:59:54.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:54.232+0000 7fc3d38da700 1 -- 172.21.15.152:0/1856365190 wait complete. 2023-12-06T03:59:54.235 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:55.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:54.231954+0000 mon.smithi132 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.152:0/1856365190' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:55.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:54.522940+0000 mon.smithi132 (mon.0) 282 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/mirror_snapshot_schedule"}]: dispatch 2023-12-06T03:59:55.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:54.530978+0000 mon.smithi132 (mon.0) 283 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/trash_purge_schedule"}]: dispatch 2023-12-06T03:59:55.616 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:55.616 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:56.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:54.451636+0000 mgr.smithi132.gokxja (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:56.669 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.667+0000 7fcc64f60700 1 -- 172.21.15.152:0/3046441967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc60060280 msgr2=0x7fcc600f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:56.669 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.667+0000 7fcc64f60700 1 --2- 172.21.15.152:0/3046441967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc60060280 0x7fcc600f90f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fcc54009fb0 tx=0x7fcc54009b30).stop 2023-12-06T03:59:56.669 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.667+0000 7fcc64f60700 1 -- 172.21.15.152:0/3046441967 shutdown_connections 2023-12-06T03:59:56.669 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.667+0000 7fcc64f60700 1 --2- 172.21.15.152:0/3046441967 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc60060280 0x7fcc600f90f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:56.669 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.667+0000 7fcc64f60700 1 -- 172.21.15.152:0/3046441967 >> 172.21.15.152:0/3046441967 conn(0x7fcc600f0b70 msgr2=0x7fcc600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:56.669 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.667+0000 7fcc64f60700 1 -- 172.21.15.152:0/3046441967 shutdown_connections 2023-12-06T03:59:56.669 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.667+0000 7fcc64f60700 1 -- 172.21.15.152:0/3046441967 wait complete. 2023-12-06T03:59:56.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.668+0000 7fcc64f60700 1 Processor -- start 2023-12-06T03:59:56.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.668+0000 7fcc64f60700 1 -- start start 2023-12-06T03:59:56.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.668+0000 7fcc64f60700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc60060280 0x7fcc601066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:56.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.668+0000 7fcc64f60700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc60106bd0 con 0x7fcc60060280 2023-12-06T03:59:56.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.668+0000 7fcc5e59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc60060280 0x7fcc601066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:56.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.668+0000 7fcc5e59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc60060280 0x7fcc601066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:49482/0 (socket says 172.21.15.152:49482) 2023-12-06T03:59:56.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.668+0000 7fcc5e59c700 1 -- 172.21.15.152:0/3628014610 learned_addr learned my addr 172.21.15.152:0/3628014610 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:56.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.668+0000 7fcc5e59c700 1 -- 172.21.15.152:0/3628014610 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc54005040 con 0x7fcc60060280 2023-12-06T03:59:56.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.668+0000 7fcc5e59c700 1 --2- 172.21.15.152:0/3628014610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc60060280 0x7fcc601066e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fcc54013510 tx=0x7fcc5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:56.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.669+0000 7fcc4effd700 1 -- 172.21.15.152:0/3628014610 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc54009320 con 0x7fcc60060280 2023-12-06T03:59:56.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.669+0000 7fcc64f60700 1 -- 172.21.15.152:0/3628014610 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc60106dd0 con 0x7fcc60060280 2023-12-06T03:59:56.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.669+0000 7fcc64f60700 1 -- 172.21.15.152:0/3628014610 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc60107260 con 0x7fcc60060280 2023-12-06T03:59:56.672 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.669+0000 7fcc4effd700 1 -- 172.21.15.152:0/3628014610 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc54013f00 con 0x7fcc60060280 2023-12-06T03:59:56.672 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.669+0000 7fcc4effd700 1 -- 172.21.15.152:0/3628014610 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc54014d90 con 0x7fcc60060280 2023-12-06T03:59:56.672 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.669+0000 7fcc4effd700 1 -- 172.21.15.152:0/3628014610 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc540263a0 con 0x7fcc60060280 2023-12-06T03:59:56.672 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.670+0000 7fcc4effd700 1 --2- 172.21.15.152:0/3628014610 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcc4802ff60 0x7fcc48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:56.672 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.670+0000 7fcc64f60700 1 -- 172.21.15.152:0/3628014610 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc44004fa0 con 0x7fcc60060280 2023-12-06T03:59:56.673 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.670+0000 7fcc4effd700 1 -- 172.21.15.152:0/3628014610 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc54041370 con 0x7fcc60060280 2023-12-06T03:59:56.673 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.670+0000 7fcc5dd9b700 1 --2- 172.21.15.152:0/3628014610 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcc4802ff60 0x7fcc48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:56.673 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.670+0000 7fcc5dd9b700 1 --2- 172.21.15.152:0/3628014610 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcc4802ff60 0x7fcc48032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fcc500096a0 tx=0x7fcc50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:56.676 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.675+0000 7fcc4effd700 1 -- 172.21.15.152:0/3628014610 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc5401d380 con 0x7fcc60060280 2023-12-06T03:59:56.885 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.883+0000 7fcc64f60700 1 -- 172.21.15.152:0/3628014610 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc44005670 con 0x7fcc60060280 2023-12-06T03:59:56.885 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.884+0000 7fcc4effd700 1 -- 172.21.15.152:0/3628014610 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcc54012330 con 0x7fcc60060280 2023-12-06T03:59:56.885 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:56.885 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:56.887 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.885+0000 7fcc64f60700 1 -- 172.21.15.152:0/3628014610 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcc4802ff60 msgr2=0x7fcc48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:56.888 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.885+0000 7fcc64f60700 1 --2- 172.21.15.152:0/3628014610 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcc4802ff60 0x7fcc48032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7fcc500096a0 tx=0x7fcc50006b40).stop 2023-12-06T03:59:56.888 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.885+0000 7fcc64f60700 1 -- 172.21.15.152:0/3628014610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc60060280 msgr2=0x7fcc601066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:56.888 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.885+0000 7fcc64f60700 1 --2- 172.21.15.152:0/3628014610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc60060280 0x7fcc601066e0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fcc54013510 tx=0x7fcc5400b040).stop 2023-12-06T03:59:56.888 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.885+0000 7fcc64f60700 1 -- 172.21.15.152:0/3628014610 shutdown_connections 2023-12-06T03:59:56.888 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.885+0000 7fcc64f60700 1 --2- 172.21.15.152:0/3628014610 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcc4802ff60 0x7fcc48032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:56.888 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.885+0000 7fcc64f60700 1 --2- 172.21.15.152:0/3628014610 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fcc60060280 0x7fcc601066e0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:56.888 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.885+0000 7fcc64f60700 1 -- 172.21.15.152:0/3628014610 >> 172.21.15.152:0/3628014610 conn(0x7fcc600f0b70 msgr2=0x7fcc600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:56.888 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.885+0000 7fcc64f60700 1 -- 172.21.15.152:0/3628014610 shutdown_connections 2023-12-06T03:59:56.888 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:56.885+0000 7fcc64f60700 1 -- 172.21.15.152:0/3628014610 wait complete. 2023-12-06T03:59:56.888 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T03:59:58.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:56.452018+0000 mgr.smithi132.gokxja (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T03:59:58.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:56.885244+0000 mon.smithi132 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.152:0/3628014610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T03:59:58.253 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T03:59:58.253 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T03:59:59.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.391+0000 7f3488d41700 1 -- 172.21.15.152:0/1989562158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34840f62a0 msgr2=0x7f34840f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:59.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.391+0000 7f3488d41700 1 --2- 172.21.15.152:0/1989562158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34840f62a0 0x7f34840f66c0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f3474009fb0 tx=0x7f3474009b30).stop 2023-12-06T03:59:59.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.391+0000 7f3488d41700 1 -- 172.21.15.152:0/1989562158 shutdown_connections 2023-12-06T03:59:59.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.391+0000 7f3488d41700 1 --2- 172.21.15.152:0/1989562158 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34840f62a0 0x7f34840f66c0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:59.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.391+0000 7f3488d41700 1 -- 172.21.15.152:0/1989562158 >> 172.21.15.152:0/1989562158 conn(0x7f34840f1aa0 msgr2=0x7f34840f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:59.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.392+0000 7f3488d41700 1 -- 172.21.15.152:0/1989562158 shutdown_connections 2023-12-06T03:59:59.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.392+0000 7f3488d41700 1 -- 172.21.15.152:0/1989562158 wait complete. 2023-12-06T03:59:59.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.392+0000 7f3488d41700 1 Processor -- start 2023-12-06T03:59:59.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.392+0000 7f3488d41700 1 -- start start 2023-12-06T03:59:59.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.392+0000 7f3488d41700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34840f62a0 0x7f34841064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:59.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.392+0000 7f3488d41700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34841069a0 con 0x7f34840f62a0 2023-12-06T03:59:59.396 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.393+0000 7f34837fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34840f62a0 0x7f34841064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:59.396 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.393+0000 7f34837fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34840f62a0 0x7f34841064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:35714/0 (socket says 172.21.15.152:35714) 2023-12-06T03:59:59.396 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.393+0000 7f34837fe700 1 -- 172.21.15.152:0/190172076 learned_addr learned my addr 172.21.15.152:0/190172076 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T03:59:59.396 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.393+0000 7f34837fe700 1 -- 172.21.15.152:0/190172076 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3474005040 con 0x7f34840f62a0 2023-12-06T03:59:59.396 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.393+0000 7f34837fe700 1 --2- 172.21.15.152:0/190172076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34840f62a0 0x7f34841064b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f3474013510 tx=0x7f347400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:59.396 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.393+0000 7f346bfff700 1 -- 172.21.15.152:0/190172076 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3474009320 con 0x7f34840f62a0 2023-12-06T03:59:59.396 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.393+0000 7f3488d41700 1 -- 172.21.15.152:0/190172076 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3484106ba0 con 0x7f34840f62a0 2023-12-06T03:59:59.396 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.393+0000 7f346bfff700 1 -- 172.21.15.152:0/190172076 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3474013f00 con 0x7f34840f62a0 2023-12-06T03:59:59.396 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.393+0000 7f346bfff700 1 -- 172.21.15.152:0/190172076 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3474014dc0 con 0x7f34840f62a0 2023-12-06T03:59:59.396 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.393+0000 7f3488d41700 1 -- 172.21.15.152:0/190172076 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3484107030 con 0x7f34840f62a0 2023-12-06T03:59:59.397 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.394+0000 7f346bfff700 1 -- 172.21.15.152:0/190172076 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f347401d3d0 con 0x7f34840f62a0 2023-12-06T03:59:59.397 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.394+0000 7f3488d41700 1 -- 172.21.15.152:0/190172076 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3484045ad0 con 0x7f34840f62a0 2023-12-06T03:59:59.397 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.394+0000 7f346bfff700 1 --2- 172.21.15.152:0/190172076 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f346c02ff60 0x7f346c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T03:59:59.397 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.394+0000 7f346bfff700 1 -- 172.21.15.152:0/190172076 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34740410a0 con 0x7f34840f62a0 2023-12-06T03:59:59.397 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.394+0000 7f3482ffd700 1 --2- 172.21.15.152:0/190172076 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f346c02ff60 0x7f346c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T03:59:59.397 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.395+0000 7f3482ffd700 1 --2- 172.21.15.152:0/190172076 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f346c02ff60 0x7f346c032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f34780096a0 tx=0x7f3478006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T03:59:59.400 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.399+0000 7f346bfff700 1 -- 172.21.15.152:0/190172076 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34740095f0 con 0x7f34840f62a0 2023-12-06T03:59:59.606 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.604+0000 7f3488d41700 1 -- 172.21.15.152:0/190172076 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3484107ee0 con 0x7f34840f62a0 2023-12-06T03:59:59.606 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.605+0000 7f346bfff700 1 -- 172.21.15.152:0/190172076 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f347401c970 con 0x7f34840f62a0 2023-12-06T03:59:59.607 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T03:59:59.607 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T03:59:59.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.606+0000 7f3488d41700 1 -- 172.21.15.152:0/190172076 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f346c02ff60 msgr2=0x7f346c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:59.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.606+0000 7f3488d41700 1 --2- 172.21.15.152:0/190172076 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f346c02ff60 0x7f346c032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f34780096a0 tx=0x7f3478006b40).stop 2023-12-06T03:59:59.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.606+0000 7f3488d41700 1 -- 172.21.15.152:0/190172076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34840f62a0 msgr2=0x7f34841064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T03:59:59.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.606+0000 7f3488d41700 1 --2- 172.21.15.152:0/190172076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34840f62a0 0x7f34841064b0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f3474013510 tx=0x7f347400b040).stop 2023-12-06T03:59:59.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.606+0000 7f3488d41700 1 -- 172.21.15.152:0/190172076 shutdown_connections 2023-12-06T03:59:59.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.606+0000 7f3488d41700 1 --2- 172.21.15.152:0/190172076 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f346c02ff60 0x7f346c032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:59.608 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.606+0000 7f3488d41700 1 --2- 172.21.15.152:0/190172076 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f34840f62a0 0x7f34841064b0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T03:59:59.609 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.606+0000 7f3488d41700 1 -- 172.21.15.152:0/190172076 >> 172.21.15.152:0/190172076 conn(0x7f34840f1aa0 msgr2=0x7f34840f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T03:59:59.609 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.606+0000 7f3488d41700 1 -- 172.21.15.152:0/190172076 shutdown_connections 2023-12-06T03:59:59.609 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T03:59:59.606+0000 7f3488d41700 1 -- 172.21.15.152:0/190172076 wait complete. 2023-12-06T03:59:59.609 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:00.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:59 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T03:59:58.452436+0000 mgr.smithi132.gokxja (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:00.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 03:59:59 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T03:59:59.606325+0000 mon.smithi132 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.152:0/190172076' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:00.933 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:00.933 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:02.055 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.051+0000 7fb9276ee700 1 -- 172.21.15.152:0/1985170721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9200f5410 msgr2=0x7fb9200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:02.055 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.051+0000 7fb9276ee700 1 --2- 172.21.15.152:0/1985170721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9200f5410 0x7fb9200f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7fb91c009fb0 tx=0x7fb91c009b30).stop 2023-12-06T04:00:02.055 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.051+0000 7fb9276ee700 1 -- 172.21.15.152:0/1985170721 shutdown_connections 2023-12-06T04:00:02.055 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.051+0000 7fb9276ee700 1 --2- 172.21.15.152:0/1985170721 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9200f5410 0x7fb9200f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:02.055 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.051+0000 7fb9276ee700 1 -- 172.21.15.152:0/1985170721 >> 172.21.15.152:0/1985170721 conn(0x7fb9200f0b70 msgr2=0x7fb9200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:02.055 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.051+0000 7fb9276ee700 1 -- 172.21.15.152:0/1985170721 shutdown_connections 2023-12-06T04:00:02.055 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.051+0000 7fb9276ee700 1 -- 172.21.15.152:0/1985170721 wait complete. 2023-12-06T04:00:02.055 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.051+0000 7fb9276ee700 1 Processor -- start 2023-12-06T04:00:02.055 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.051+0000 7fb9276ee700 1 -- start start 2023-12-06T04:00:02.055 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.052+0000 7fb9276ee700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9200f5410 0x7fb920105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:02.055 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.052+0000 7fb9276ee700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb920105b20 con 0x7fb9200f5410 2023-12-06T04:00:02.055 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.052+0000 7fb92548a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9200f5410 0x7fb920105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:02.056 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.052+0000 7fb92548a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9200f5410 0x7fb920105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:35722/0 (socket says 172.21.15.152:35722) 2023-12-06T04:00:02.056 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.052+0000 7fb92548a700 1 -- 172.21.15.152:0/2022270054 learned_addr learned my addr 172.21.15.152:0/2022270054 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:02.056 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.052+0000 7fb92548a700 1 -- 172.21.15.152:0/2022270054 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb91c005040 con 0x7fb9200f5410 2023-12-06T04:00:02.056 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.052+0000 7fb92548a700 1 --2- 172.21.15.152:0/2022270054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9200f5410 0x7fb920105630 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fb91c013510 tx=0x7fb91c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:02.056 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.052+0000 7fb911ffb700 1 -- 172.21.15.152:0/2022270054 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb91c009320 con 0x7fb9200f5410 2023-12-06T04:00:02.056 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.052+0000 7fb9276ee700 1 -- 172.21.15.152:0/2022270054 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb920105d20 con 0x7fb9200f5410 2023-12-06T04:00:02.056 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.052+0000 7fb911ffb700 1 -- 172.21.15.152:0/2022270054 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb91c013f00 con 0x7fb9200f5410 2023-12-06T04:00:02.056 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.052+0000 7fb911ffb700 1 -- 172.21.15.152:0/2022270054 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb91c014d90 con 0x7fb9200f5410 2023-12-06T04:00:02.056 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.053+0000 7fb9276ee700 1 -- 172.21.15.152:0/2022270054 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9201061b0 con 0x7fb9200f5410 2023-12-06T04:00:02.057 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.053+0000 7fb911ffb700 1 -- 172.21.15.152:0/2022270054 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb91c0095f0 con 0x7fb9200f5410 2023-12-06T04:00:02.057 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.053+0000 7fb9276ee700 1 -- 172.21.15.152:0/2022270054 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb920045ad0 con 0x7fb9200f5410 2023-12-06T04:00:02.057 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.054+0000 7fb911ffb700 1 --2- 172.21.15.152:0/2022270054 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb90c02ff60 0x7fb90c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:02.057 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.054+0000 7fb911ffb700 1 -- 172.21.15.152:0/2022270054 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb91c041000 con 0x7fb9200f5410 2023-12-06T04:00:02.057 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.054+0000 7fb924c89700 1 --2- 172.21.15.152:0/2022270054 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb90c02ff60 0x7fb90c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:02.057 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.054+0000 7fb924c89700 1 --2- 172.21.15.152:0/2022270054 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb90c02ff60 0x7fb90c032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb9140096a0 tx=0x7fb914006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:02.059 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.058+0000 7fb911ffb700 1 -- 172.21.15.152:0/2022270054 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb91c01d7f0 con 0x7fb9200f5410 2023-12-06T04:00:02.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:01 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:00.452831+0000 mgr.smithi132.gokxja (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:02.266 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.264+0000 7fb9276ee700 1 -- 172.21.15.152:0/2022270054 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9200fae60 con 0x7fb9200f5410 2023-12-06T04:00:02.267 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.266+0000 7fb911ffb700 1 -- 172.21.15.152:0/2022270054 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb91c010230 con 0x7fb9200f5410 2023-12-06T04:00:02.267 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:02.267 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:02.268 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.267+0000 7fb9276ee700 1 -- 172.21.15.152:0/2022270054 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb90c02ff60 msgr2=0x7fb90c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:02.268 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.267+0000 7fb9276ee700 1 --2- 172.21.15.152:0/2022270054 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb90c02ff60 0x7fb90c032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fb9140096a0 tx=0x7fb914006b40).stop 2023-12-06T04:00:02.269 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.267+0000 7fb9276ee700 1 -- 172.21.15.152:0/2022270054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9200f5410 msgr2=0x7fb920105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:02.269 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.267+0000 7fb9276ee700 1 --2- 172.21.15.152:0/2022270054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9200f5410 0x7fb920105630 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7fb91c013510 tx=0x7fb91c00b040).stop 2023-12-06T04:00:02.269 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.267+0000 7fb9276ee700 1 -- 172.21.15.152:0/2022270054 shutdown_connections 2023-12-06T04:00:02.269 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.267+0000 7fb9276ee700 1 --2- 172.21.15.152:0/2022270054 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb90c02ff60 0x7fb90c032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:02.269 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.267+0000 7fb9276ee700 1 --2- 172.21.15.152:0/2022270054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb9200f5410 0x7fb920105630 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:02.269 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.267+0000 7fb9276ee700 1 -- 172.21.15.152:0/2022270054 >> 172.21.15.152:0/2022270054 conn(0x7fb9200f0b70 msgr2=0x7fb9200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:02.269 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.267+0000 7fb9276ee700 1 -- 172.21.15.152:0/2022270054 shutdown_connections 2023-12-06T04:00:02.269 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:02.267+0000 7fb9276ee700 1 -- 172.21.15.152:0/2022270054 wait complete. 2023-12-06T04:00:02.269 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:03.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:02.266535+0000 mon.smithi132 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.152:0/2022270054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:03.637 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:03.637 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:04.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:02.453193+0000 mgr.smithi132.gokxja (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:04.762 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.759+0000 7f7f1f745700 1 -- 172.21.15.152:0/2448515568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f180f5410 msgr2=0x7f7f180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:04.762 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.759+0000 7f7f1f745700 1 --2- 172.21.15.152:0/2448515568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f180f5410 0x7f7f180f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f7f08009fb0 tx=0x7f7f08009b30).stop 2023-12-06T04:00:04.762 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.759+0000 7f7f1f745700 1 -- 172.21.15.152:0/2448515568 shutdown_connections 2023-12-06T04:00:04.762 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.759+0000 7f7f1f745700 1 --2- 172.21.15.152:0/2448515568 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f180f5410 0x7f7f180f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:04.762 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.759+0000 7f7f1f745700 1 -- 172.21.15.152:0/2448515568 >> 172.21.15.152:0/2448515568 conn(0x7f7f180f0b70 msgr2=0x7f7f180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:04.762 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.759+0000 7f7f1f745700 1 -- 172.21.15.152:0/2448515568 shutdown_connections 2023-12-06T04:00:04.763 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.759+0000 7f7f1f745700 1 -- 172.21.15.152:0/2448515568 wait complete. 2023-12-06T04:00:04.763 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.760+0000 7f7f1f745700 1 Processor -- start 2023-12-06T04:00:04.763 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.760+0000 7f7f1f745700 1 -- start start 2023-12-06T04:00:04.763 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.760+0000 7f7f1f745700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f180f5410 0x7f7f180fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:04.763 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.760+0000 7f7f1f745700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f180fb820 con 0x7f7f180f5410 2023-12-06T04:00:04.763 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.760+0000 7f7f1d4e1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f180f5410 0x7f7f180fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:04.763 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.760+0000 7f7f1d4e1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f180f5410 0x7f7f180fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:35734/0 (socket says 172.21.15.152:35734) 2023-12-06T04:00:04.763 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.760+0000 7f7f1d4e1700 1 -- 172.21.15.152:0/171198003 learned_addr learned my addr 172.21.15.152:0/171198003 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:04.764 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.761+0000 7f7f1d4e1700 1 -- 172.21.15.152:0/171198003 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f08005040 con 0x7f7f180f5410 2023-12-06T04:00:04.764 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.761+0000 7f7f1d4e1700 1 --2- 172.21.15.152:0/171198003 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f180f5410 0x7f7f180fd3a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f7f08014000 tx=0x7f7f0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:04.765 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.761+0000 7f7f0dffb700 1 -- 172.21.15.152:0/171198003 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f080095f0 con 0x7f7f180f5410 2023-12-06T04:00:04.765 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.761+0000 7f7f1f745700 1 -- 172.21.15.152:0/171198003 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f180fb9c0 con 0x7f7f180f5410 2023-12-06T04:00:04.765 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.761+0000 7f7f0dffb700 1 -- 172.21.15.152:0/171198003 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f08007e70 con 0x7f7f180f5410 2023-12-06T04:00:04.765 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.761+0000 7f7f0dffb700 1 -- 172.21.15.152:0/171198003 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f08014ea0 con 0x7f7f180f5410 2023-12-06T04:00:04.765 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.761+0000 7f7f1f745700 1 -- 172.21.15.152:0/171198003 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f180fbe50 con 0x7f7f180f5410 2023-12-06T04:00:04.766 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.762+0000 7f7f0dffb700 1 -- 172.21.15.152:0/171198003 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f08025070 con 0x7f7f180f5410 2023-12-06T04:00:04.766 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.762+0000 7f7f1f745700 1 -- 172.21.15.152:0/171198003 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f18059100 con 0x7f7f180f5410 2023-12-06T04:00:04.766 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.762+0000 7f7f0dffb700 1 --2- 172.21.15.152:0/171198003 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7f0402ffb0 0x7f7f04032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:04.766 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.762+0000 7f7f0dffb700 1 -- 172.21.15.152:0/171198003 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f080409e0 con 0x7f7f180f5410 2023-12-06T04:00:04.766 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.762+0000 7f7f1cce0700 1 --2- 172.21.15.152:0/171198003 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7f0402ffb0 0x7f7f04032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:04.766 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.763+0000 7f7f1cce0700 1 --2- 172.21.15.152:0/171198003 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7f0402ffb0 0x7f7f04032470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f7f140096a0 tx=0x7f7f14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:04.768 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.767+0000 7f7f0dffb700 1 -- 172.21.15.152:0/171198003 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f0800f300 con 0x7f7f180f5410 2023-12-06T04:00:04.975 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.973+0000 7f7f1f745700 1 -- 172.21.15.152:0/171198003 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f180fae60 con 0x7f7f180f5410 2023-12-06T04:00:04.975 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.974+0000 7f7f0dffb700 1 -- 172.21.15.152:0/171198003 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7f0800f300 con 0x7f7f180f5410 2023-12-06T04:00:04.976 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:04.976 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:04.977 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.975+0000 7f7f1f745700 1 -- 172.21.15.152:0/171198003 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7f0402ffb0 msgr2=0x7f7f04032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:04.977 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.975+0000 7f7f1f745700 1 --2- 172.21.15.152:0/171198003 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7f0402ffb0 0x7f7f04032470 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f7f140096a0 tx=0x7f7f14006b40).stop 2023-12-06T04:00:04.977 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.975+0000 7f7f1f745700 1 -- 172.21.15.152:0/171198003 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f180f5410 msgr2=0x7f7f180fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:04.977 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.975+0000 7f7f1f745700 1 --2- 172.21.15.152:0/171198003 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f180f5410 0x7f7f180fd3a0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f7f08014000 tx=0x7f7f0800b040).stop 2023-12-06T04:00:04.977 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.975+0000 7f7f1f745700 1 -- 172.21.15.152:0/171198003 shutdown_connections 2023-12-06T04:00:04.978 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.975+0000 7f7f1f745700 1 --2- 172.21.15.152:0/171198003 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7f0402ffb0 0x7f7f04032470 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:04.978 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.975+0000 7f7f1f745700 1 --2- 172.21.15.152:0/171198003 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7f180f5410 0x7f7f180fd3a0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:04.978 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.975+0000 7f7f1f745700 1 -- 172.21.15.152:0/171198003 >> 172.21.15.152:0/171198003 conn(0x7f7f180f0b70 msgr2=0x7f7f1805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:04.978 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.975+0000 7f7f1f745700 1 -- 172.21.15.152:0/171198003 shutdown_connections 2023-12-06T04:00:04.978 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:04.975+0000 7f7f1f745700 1 -- 172.21.15.152:0/171198003 wait complete. 2023-12-06T04:00:04.978 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:06.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:04.453538+0000 mgr.smithi132.gokxja (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:06.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:04.975247+0000 mon.smithi132 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.152:0/171198003' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:06.364 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:06.364 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:07.459 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.454+0000 7f86b3525700 1 -- 172.21.15.152:0/3160377250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86ac0f5410 msgr2=0x7f86ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:07.460 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.454+0000 7f86b3525700 1 --2- 172.21.15.152:0/3160377250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86ac0f5410 0x7f86ac0f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f869c009fb0 tx=0x7f869c009b30).stop 2023-12-06T04:00:07.460 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.454+0000 7f86b3525700 1 -- 172.21.15.152:0/3160377250 shutdown_connections 2023-12-06T04:00:07.460 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.454+0000 7f86b3525700 1 --2- 172.21.15.152:0/3160377250 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86ac0f5410 0x7f86ac0f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:07.460 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.454+0000 7f86b3525700 1 -- 172.21.15.152:0/3160377250 >> 172.21.15.152:0/3160377250 conn(0x7f86ac0f0b70 msgr2=0x7f86ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:07.460 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.454+0000 7f86b3525700 1 -- 172.21.15.152:0/3160377250 shutdown_connections 2023-12-06T04:00:07.460 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.454+0000 7f86b3525700 1 -- 172.21.15.152:0/3160377250 wait complete. 2023-12-06T04:00:07.460 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.455+0000 7f86b3525700 1 Processor -- start 2023-12-06T04:00:07.460 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.455+0000 7f86b3525700 1 -- start start 2023-12-06T04:00:07.460 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.455+0000 7f86b3525700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86ac0f5410 0x7f86ac10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:07.460 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.455+0000 7f86b3525700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac107240 con 0x7f86ac0f5410 2023-12-06T04:00:07.460 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.455+0000 7f86b12c1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86ac0f5410 0x7f86ac10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:07.460 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.455+0000 7f86b12c1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86ac0f5410 0x7f86ac10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:35754/0 (socket says 172.21.15.152:35754) 2023-12-06T04:00:07.461 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.455+0000 7f86b12c1700 1 -- 172.21.15.152:0/2444894414 learned_addr learned my addr 172.21.15.152:0/2444894414 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:07.461 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.455+0000 7f86b12c1700 1 -- 172.21.15.152:0/2444894414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f869c005040 con 0x7f86ac0f5410 2023-12-06T04:00:07.461 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.456+0000 7f86b12c1700 1 --2- 172.21.15.152:0/2444894414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86ac0f5410 0x7f86ac10b850 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f869c013510 tx=0x7f869c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:07.461 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.456+0000 7f86a1ffb700 1 -- 172.21.15.152:0/2444894414 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f869c009320 con 0x7f86ac0f5410 2023-12-06T04:00:07.461 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.456+0000 7f86b3525700 1 -- 172.21.15.152:0/2444894414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ac060220 con 0x7f86ac0f5410 2023-12-06T04:00:07.461 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.456+0000 7f86a1ffb700 1 -- 172.21.15.152:0/2444894414 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f869c013f00 con 0x7f86ac0f5410 2023-12-06T04:00:07.461 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.456+0000 7f86a1ffb700 1 -- 172.21.15.152:0/2444894414 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f869c014d90 con 0x7f86ac0f5410 2023-12-06T04:00:07.461 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.456+0000 7f86b3525700 1 -- 172.21.15.152:0/2444894414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ac10bd40 con 0x7f86ac0f5410 2023-12-06T04:00:07.461 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.457+0000 7f86a1ffb700 1 -- 172.21.15.152:0/2444894414 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f869c0095f0 con 0x7f86ac0f5410 2023-12-06T04:00:07.461 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.457+0000 7f86b3525700 1 -- 172.21.15.152:0/2444894414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ac045ad0 con 0x7f86ac0f5410 2023-12-06T04:00:07.461 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.457+0000 7f86a1ffb700 1 --2- 172.21.15.152:0/2444894414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f869802ff60 0x7f8698032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:07.461 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.457+0000 7f86a1ffb700 1 -- 172.21.15.152:0/2444894414 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f869c041000 con 0x7f86ac0f5410 2023-12-06T04:00:07.462 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.457+0000 7f86b0ac0700 1 --2- 172.21.15.152:0/2444894414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f869802ff60 0x7f8698032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:07.462 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.457+0000 7f86b0ac0700 1 --2- 172.21.15.152:0/2444894414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f869802ff60 0x7f8698032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f86a80096a0 tx=0x7f86a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:07.464 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.461+0000 7f86a1ffb700 1 -- 172.21.15.152:0/2444894414 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f869c01d7f0 con 0x7f86ac0f5410 2023-12-06T04:00:07.670 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.668+0000 7f86b3525700 1 -- 172.21.15.152:0/2444894414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86ac0fae60 con 0x7f86ac0f5410 2023-12-06T04:00:07.671 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.669+0000 7f86a1ffb700 1 -- 172.21.15.152:0/2444894414 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f869c010230 con 0x7f86ac0f5410 2023-12-06T04:00:07.671 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:07.671 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:07.673 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.671+0000 7f86b3525700 1 -- 172.21.15.152:0/2444894414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f869802ff60 msgr2=0x7f8698032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:07.673 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.671+0000 7f86b3525700 1 --2- 172.21.15.152:0/2444894414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f869802ff60 0x7f8698032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f86a80096a0 tx=0x7f86a8006b40).stop 2023-12-06T04:00:07.673 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.671+0000 7f86b3525700 1 -- 172.21.15.152:0/2444894414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86ac0f5410 msgr2=0x7f86ac10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:07.673 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.671+0000 7f86b3525700 1 --2- 172.21.15.152:0/2444894414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86ac0f5410 0x7f86ac10b850 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f869c013510 tx=0x7f869c00b040).stop 2023-12-06T04:00:07.673 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.671+0000 7f86b3525700 1 -- 172.21.15.152:0/2444894414 shutdown_connections 2023-12-06T04:00:07.673 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.671+0000 7f86b3525700 1 --2- 172.21.15.152:0/2444894414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f869802ff60 0x7f8698032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:07.673 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.671+0000 7f86b3525700 1 --2- 172.21.15.152:0/2444894414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f86ac0f5410 0x7f86ac10b850 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:07.673 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.671+0000 7f86b3525700 1 -- 172.21.15.152:0/2444894414 >> 172.21.15.152:0/2444894414 conn(0x7f86ac0f0b70 msgr2=0x7f86ac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:07.674 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.671+0000 7f86b3525700 1 -- 172.21.15.152:0/2444894414 shutdown_connections 2023-12-06T04:00:07.674 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:07.671+0000 7f86b3525700 1 -- 172.21.15.152:0/2444894414 wait complete. 2023-12-06T04:00:07.674 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:08.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:07 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:06.453806+0000 mgr.smithi132.gokxja (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:09.029 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:09.030 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:09.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:08 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:07.670191+0000 mon.smithi132 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.152:0/2444894414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:10.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:09 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:08.454177+0000 mgr.smithi132.gokxja (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:10.135 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.132+0000 7f4d9050a700 1 -- 172.21.15.152:0/1704948206 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d880f5410 msgr2=0x7f4d880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.132+0000 7f4d9050a700 1 --2- 172.21.15.152:0/1704948206 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d880f5410 0x7f4d880f5830 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f4d7c009fb0 tx=0x7f4d7c009b30).stop 2023-12-06T04:00:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.132+0000 7f4d9050a700 1 -- 172.21.15.152:0/1704948206 shutdown_connections 2023-12-06T04:00:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.132+0000 7f4d9050a700 1 --2- 172.21.15.152:0/1704948206 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d880f5410 0x7f4d880f5830 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.132+0000 7f4d9050a700 1 -- 172.21.15.152:0/1704948206 >> 172.21.15.152:0/1704948206 conn(0x7f4d880f0b70 msgr2=0x7f4d880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.132+0000 7f4d9050a700 1 -- 172.21.15.152:0/1704948206 shutdown_connections 2023-12-06T04:00:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.132+0000 7f4d9050a700 1 -- 172.21.15.152:0/1704948206 wait complete. 2023-12-06T04:00:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.133+0000 7f4d9050a700 1 Processor -- start 2023-12-06T04:00:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.133+0000 7f4d9050a700 1 -- start start 2023-12-06T04:00:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.133+0000 7f4d9050a700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d880f5410 0x7f4d88105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.133+0000 7f4d9050a700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d88105b20 con 0x7f4d880f5410 2023-12-06T04:00:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.133+0000 7f4d8e2a6700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d880f5410 0x7f4d88105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.133+0000 7f4d8e2a6700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d880f5410 0x7f4d88105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:54564/0 (socket says 172.21.15.152:54564) 2023-12-06T04:00:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.133+0000 7f4d8e2a6700 1 -- 172.21.15.152:0/544726229 learned_addr learned my addr 172.21.15.152:0/544726229 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:10.136 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.133+0000 7f4d8e2a6700 1 -- 172.21.15.152:0/544726229 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7c005040 con 0x7f4d880f5410 2023-12-06T04:00:10.137 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.133+0000 7f4d8e2a6700 1 --2- 172.21.15.152:0/544726229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d880f5410 0x7f4d88105630 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f4d7c013510 tx=0x7f4d7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:10.137 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.134+0000 7f4d7affd700 1 -- 172.21.15.152:0/544726229 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d7c009320 con 0x7f4d880f5410 2023-12-06T04:00:10.137 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.134+0000 7f4d9050a700 1 -- 172.21.15.152:0/544726229 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d88105d20 con 0x7f4d880f5410 2023-12-06T04:00:10.137 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.134+0000 7f4d9050a700 1 -- 172.21.15.152:0/544726229 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d881061b0 con 0x7f4d880f5410 2023-12-06T04:00:10.137 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.134+0000 7f4d7affd700 1 -- 172.21.15.152:0/544726229 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d7c013f00 con 0x7f4d880f5410 2023-12-06T04:00:10.137 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.134+0000 7f4d7affd700 1 -- 172.21.15.152:0/544726229 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d7c014ce0 con 0x7f4d880f5410 2023-12-06T04:00:10.138 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.135+0000 7f4d7affd700 1 -- 172.21.15.152:0/544726229 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d7c014420 con 0x7f4d880f5410 2023-12-06T04:00:10.138 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.135+0000 7f4d9050a700 1 -- 172.21.15.152:0/544726229 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d70004fa0 con 0x7f4d880f5410 2023-12-06T04:00:10.138 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.135+0000 7f4d7affd700 1 --2- 172.21.15.152:0/544726229 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4d7402ff60 0x7f4d74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:10.138 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.135+0000 7f4d7affd700 1 -- 172.21.15.152:0/544726229 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d7c0410a0 con 0x7f4d880f5410 2023-12-06T04:00:10.138 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.135+0000 7f4d8daa5700 1 --2- 172.21.15.152:0/544726229 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4d7402ff60 0x7f4d74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:10.138 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.135+0000 7f4d8daa5700 1 --2- 172.21.15.152:0/544726229 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4d7402ff60 0x7f4d74032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4d840096a0 tx=0x7f4d84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:10.141 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.139+0000 7f4d7affd700 1 -- 172.21.15.152:0/544726229 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d7c012930 con 0x7f4d880f5410 2023-12-06T04:00:10.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.347+0000 7f4d9050a700 1 -- 172.21.15.152:0/544726229 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d70005670 con 0x7f4d880f5410 2023-12-06T04:00:10.349 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.347+0000 7f4d7affd700 1 -- 172.21.15.152:0/544726229 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4d7c012930 con 0x7f4d880f5410 2023-12-06T04:00:10.350 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:10.350 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:10.351 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.349+0000 7f4d9050a700 1 -- 172.21.15.152:0/544726229 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4d7402ff60 msgr2=0x7f4d74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:10.351 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.349+0000 7f4d9050a700 1 --2- 172.21.15.152:0/544726229 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4d7402ff60 0x7f4d74032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f4d840096a0 tx=0x7f4d84006b40).stop 2023-12-06T04:00:10.351 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.349+0000 7f4d9050a700 1 -- 172.21.15.152:0/544726229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d880f5410 msgr2=0x7f4d88105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:10.351 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.349+0000 7f4d9050a700 1 --2- 172.21.15.152:0/544726229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d880f5410 0x7f4d88105630 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f4d7c013510 tx=0x7f4d7c00b040).stop 2023-12-06T04:00:10.351 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.349+0000 7f4d9050a700 1 -- 172.21.15.152:0/544726229 shutdown_connections 2023-12-06T04:00:10.352 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.349+0000 7f4d9050a700 1 --2- 172.21.15.152:0/544726229 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f4d7402ff60 0x7f4d74032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:10.352 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.349+0000 7f4d9050a700 1 --2- 172.21.15.152:0/544726229 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f4d880f5410 0x7f4d88105630 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:10.352 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.349+0000 7f4d9050a700 1 -- 172.21.15.152:0/544726229 >> 172.21.15.152:0/544726229 conn(0x7f4d880f0b70 msgr2=0x7f4d880f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:10.352 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.349+0000 7f4d9050a700 1 -- 172.21.15.152:0/544726229 shutdown_connections 2023-12-06T04:00:10.352 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:10.349+0000 7f4d9050a700 1 -- 172.21.15.152:0/544726229 wait complete. 2023-12-06T04:00:10.352 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:11.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:10 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:10.348785+0000 mon.smithi132 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.152:0/544726229' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:11.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:11.735 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:12.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:11 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:10.454546+0000 mgr.smithi132.gokxja (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:12.814 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.812+0000 7fd07693f700 1 -- 172.21.15.152:0/1248434290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0700f5410 msgr2=0x7fd0700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:12.815 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.812+0000 7fd07693f700 1 --2- 172.21.15.152:0/1248434290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0700f5410 0x7fd0700f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7fd064009fb0 tx=0x7fd064009b30).stop 2023-12-06T04:00:12.815 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.812+0000 7fd07693f700 1 -- 172.21.15.152:0/1248434290 shutdown_connections 2023-12-06T04:00:12.815 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.812+0000 7fd07693f700 1 --2- 172.21.15.152:0/1248434290 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0700f5410 0x7fd0700f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:12.815 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.812+0000 7fd07693f700 1 -- 172.21.15.152:0/1248434290 >> 172.21.15.152:0/1248434290 conn(0x7fd0700f0b70 msgr2=0x7fd0700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:12.815 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.812+0000 7fd07693f700 1 -- 172.21.15.152:0/1248434290 shutdown_connections 2023-12-06T04:00:12.815 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.813+0000 7fd07693f700 1 -- 172.21.15.152:0/1248434290 wait complete. 2023-12-06T04:00:12.815 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.813+0000 7fd07693f700 1 Processor -- start 2023-12-06T04:00:12.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.813+0000 7fd07693f700 1 -- start start 2023-12-06T04:00:12.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.813+0000 7fd07693f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0700f5410 0x7fd070109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:12.816 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.813+0000 7fd07693f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd070060220 con 0x7fd0700f5410 2023-12-06T04:00:12.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.813+0000 7fd06ffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0700f5410 0x7fd070109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:12.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.814+0000 7fd06ffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0700f5410 0x7fd070109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:54572/0 (socket says 172.21.15.152:54572) 2023-12-06T04:00:12.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.814+0000 7fd06ffff700 1 -- 172.21.15.152:0/4284251054 learned_addr learned my addr 172.21.15.152:0/4284251054 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:12.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.814+0000 7fd06ffff700 1 -- 172.21.15.152:0/4284251054 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd064005040 con 0x7fd0700f5410 2023-12-06T04:00:12.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.814+0000 7fd06ffff700 1 --2- 172.21.15.152:0/4284251054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0700f5410 0x7fd070109450 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fd064013510 tx=0x7fd06400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:12.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.814+0000 7fd06cff9700 1 -- 172.21.15.152:0/4284251054 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd064009320 con 0x7fd0700f5410 2023-12-06T04:00:12.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.814+0000 7fd07693f700 1 -- 172.21.15.152:0/4284251054 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd070060420 con 0x7fd0700f5410 2023-12-06T04:00:12.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.814+0000 7fd06cff9700 1 -- 172.21.15.152:0/4284251054 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd064013f00 con 0x7fd0700f5410 2023-12-06T04:00:12.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.814+0000 7fd06cff9700 1 -- 172.21.15.152:0/4284251054 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd064014d90 con 0x7fd0700f5410 2023-12-06T04:00:12.817 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.814+0000 7fd07693f700 1 -- 172.21.15.152:0/4284251054 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd070109b00 con 0x7fd0700f5410 2023-12-06T04:00:12.818 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.815+0000 7fd06cff9700 1 -- 172.21.15.152:0/4284251054 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0640263a0 con 0x7fd0700f5410 2023-12-06T04:00:12.818 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.815+0000 7fd07693f700 1 -- 172.21.15.152:0/4284251054 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd070059100 con 0x7fd0700f5410 2023-12-06T04:00:12.818 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.815+0000 7fd06cff9700 1 --2- 172.21.15.152:0/4284251054 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd05802ff60 0x7fd058032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:12.818 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.815+0000 7fd06cff9700 1 -- 172.21.15.152:0/4284251054 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd064041310 con 0x7fd0700f5410 2023-12-06T04:00:12.819 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.815+0000 7fd06f7fe700 1 --2- 172.21.15.152:0/4284251054 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd05802ff60 0x7fd058032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:12.819 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.816+0000 7fd06f7fe700 1 --2- 172.21.15.152:0/4284251054 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd05802ff60 0x7fd058032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fd0600096a0 tx=0x7fd060006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:12.822 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:12.820+0000 7fd06cff9700 1 -- 172.21.15.152:0/4284251054 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd064012330 con 0x7fd0700f5410 2023-12-06T04:00:13.029 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:13.028+0000 7fd07693f700 1 -- 172.21.15.152:0/4284251054 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd070045ad0 con 0x7fd0700f5410 2023-12-06T04:00:13.030 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:13.028+0000 7fd06cff9700 1 -- 172.21.15.152:0/4284251054 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd064014420 con 0x7fd0700f5410 2023-12-06T04:00:13.030 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:13.031 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:13.032 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:13.030+0000 7fd07693f700 1 -- 172.21.15.152:0/4284251054 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd05802ff60 msgr2=0x7fd058032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:13.032 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:13.030+0000 7fd07693f700 1 --2- 172.21.15.152:0/4284251054 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd05802ff60 0x7fd058032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fd0600096a0 tx=0x7fd060006b40).stop 2023-12-06T04:00:13.032 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:13.030+0000 7fd07693f700 1 -- 172.21.15.152:0/4284251054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0700f5410 msgr2=0x7fd070109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:13.032 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:13.030+0000 7fd07693f700 1 --2- 172.21.15.152:0/4284251054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0700f5410 0x7fd070109450 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fd064013510 tx=0x7fd06400b040).stop 2023-12-06T04:00:13.032 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:13.030+0000 7fd07693f700 1 -- 172.21.15.152:0/4284251054 shutdown_connections 2023-12-06T04:00:13.032 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:13.030+0000 7fd07693f700 1 --2- 172.21.15.152:0/4284251054 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd05802ff60 0x7fd058032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:13.032 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:13.030+0000 7fd07693f700 1 --2- 172.21.15.152:0/4284251054 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd0700f5410 0x7fd070109450 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:13.033 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:13.030+0000 7fd07693f700 1 -- 172.21.15.152:0/4284251054 >> 172.21.15.152:0/4284251054 conn(0x7fd0700f0b70 msgr2=0x7fd0700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:13.033 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:13.030+0000 7fd07693f700 1 -- 172.21.15.152:0/4284251054 shutdown_connections 2023-12-06T04:00:13.033 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:13.030+0000 7fd07693f700 1 -- 172.21.15.152:0/4284251054 wait complete. 2023-12-06T04:00:13.033 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:14.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:13 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:12.454928+0000 mgr.smithi132.gokxja (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:14.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:13 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:13.029871+0000 mon.smithi132 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.152:0/4284251054' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:14.405 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:14.405 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:15.508 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.502+0000 7f7faf1c0700 1 -- 172.21.15.152:0/1825988813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fa80f51a0 msgr2=0x7f7fa80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:15.508 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.502+0000 7f7faf1c0700 1 --2- 172.21.15.152:0/1825988813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fa80f51a0 0x7f7fa80f55c0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f7f98009fb0 tx=0x7f7f98009b30).stop 2023-12-06T04:00:15.508 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.503+0000 7f7faf1c0700 1 -- 172.21.15.152:0/1825988813 shutdown_connections 2023-12-06T04:00:15.509 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.503+0000 7f7faf1c0700 1 --2- 172.21.15.152:0/1825988813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fa80f51a0 0x7f7fa80f55c0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:15.509 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.503+0000 7f7faf1c0700 1 -- 172.21.15.152:0/1825988813 >> 172.21.15.152:0/1825988813 conn(0x7f7fa80f09a0 msgr2=0x7f7fa80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:15.509 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.503+0000 7f7faf1c0700 1 -- 172.21.15.152:0/1825988813 shutdown_connections 2023-12-06T04:00:15.509 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.503+0000 7f7faf1c0700 1 -- 172.21.15.152:0/1825988813 wait complete. 2023-12-06T04:00:15.509 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.503+0000 7f7faf1c0700 1 Processor -- start 2023-12-06T04:00:15.509 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.503+0000 7f7faf1c0700 1 -- start start 2023-12-06T04:00:15.509 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.503+0000 7f7faf1c0700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fa80f51a0 0x7f7fa81086c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:15.510 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.503+0000 7f7faf1c0700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fa8108bb0 con 0x7f7fa80f51a0 2023-12-06T04:00:15.510 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.503+0000 7f7fae1be700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fa80f51a0 0x7f7fa81086c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:15.510 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.503+0000 7f7fae1be700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fa80f51a0 0x7f7fa81086c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:54580/0 (socket says 172.21.15.152:54580) 2023-12-06T04:00:15.510 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.503+0000 7f7fae1be700 1 -- 172.21.15.152:0/3325425877 learned_addr learned my addr 172.21.15.152:0/3325425877 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:15.510 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.504+0000 7f7fae1be700 1 -- 172.21.15.152:0/3325425877 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f98005040 con 0x7f7fa80f51a0 2023-12-06T04:00:15.510 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.504+0000 7f7fae1be700 1 --2- 172.21.15.152:0/3325425877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fa80f51a0 0x7f7fa81086c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f7f98013510 tx=0x7f7f9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:15.510 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.504+0000 7f7f9effd700 1 -- 172.21.15.152:0/3325425877 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f98009320 con 0x7f7fa80f51a0 2023-12-06T04:00:15.511 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.504+0000 7f7faf1c0700 1 -- 172.21.15.152:0/3325425877 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fa8108db0 con 0x7f7fa80f51a0 2023-12-06T04:00:15.511 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.504+0000 7f7f9effd700 1 -- 172.21.15.152:0/3325425877 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f98013f00 con 0x7f7fa80f51a0 2023-12-06T04:00:15.511 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.504+0000 7f7f9effd700 1 -- 172.21.15.152:0/3325425877 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f98014dc0 con 0x7f7fa80f51a0 2023-12-06T04:00:15.511 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.504+0000 7f7faf1c0700 1 -- 172.21.15.152:0/3325425877 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fa8109240 con 0x7f7fa80f51a0 2023-12-06T04:00:15.511 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.505+0000 7f7f9effd700 1 -- 172.21.15.152:0/3325425877 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f980095f0 con 0x7f7fa80f51a0 2023-12-06T04:00:15.511 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.505+0000 7f7faf1c0700 1 -- 172.21.15.152:0/3325425877 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fa8045ad0 con 0x7f7fa80f51a0 2023-12-06T04:00:15.512 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.505+0000 7f7f9effd700 1 --2- 172.21.15.152:0/3325425877 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7f9402ff60 0x7f7f94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:15.512 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.505+0000 7f7f9effd700 1 -- 172.21.15.152:0/3325425877 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f980410e0 con 0x7f7fa80f51a0 2023-12-06T04:00:15.512 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.505+0000 7f7fad9bd700 1 --2- 172.21.15.152:0/3325425877 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7f9402ff60 0x7f7f94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:15.512 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.506+0000 7f7fad9bd700 1 --2- 172.21.15.152:0/3325425877 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7f9402ff60 0x7f7f94032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7fa40096a0 tx=0x7f7fa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:15.513 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.509+0000 7f7f9effd700 1 -- 172.21.15.152:0/3325425877 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f98014420 con 0x7f7fa80f51a0 2023-12-06T04:00:15.718 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.716+0000 7f7faf1c0700 1 -- 172.21.15.152:0/3325425877 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7fa810a090 con 0x7f7fa80f51a0 2023-12-06T04:00:15.719 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.717+0000 7f7f9effd700 1 -- 172.21.15.152:0/3325425877 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7f9801c920 con 0x7f7fa80f51a0 2023-12-06T04:00:15.719 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:15.719 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:15.721 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.719+0000 7f7faf1c0700 1 -- 172.21.15.152:0/3325425877 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7f9402ff60 msgr2=0x7f7f94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:15.721 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.719+0000 7f7faf1c0700 1 --2- 172.21.15.152:0/3325425877 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7f9402ff60 0x7f7f94032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7fa40096a0 tx=0x7f7fa4006b40).stop 2023-12-06T04:00:15.721 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.719+0000 7f7faf1c0700 1 -- 172.21.15.152:0/3325425877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fa80f51a0 msgr2=0x7f7fa81086c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:15.721 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.719+0000 7f7faf1c0700 1 --2- 172.21.15.152:0/3325425877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fa80f51a0 0x7f7fa81086c0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f7f98013510 tx=0x7f7f9800b040).stop 2023-12-06T04:00:15.721 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.719+0000 7f7faf1c0700 1 -- 172.21.15.152:0/3325425877 shutdown_connections 2023-12-06T04:00:15.721 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.719+0000 7f7faf1c0700 1 --2- 172.21.15.152:0/3325425877 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f7f9402ff60 0x7f7f94032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:15.721 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.719+0000 7f7faf1c0700 1 --2- 172.21.15.152:0/3325425877 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f7fa80f51a0 0x7f7fa81086c0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:15.721 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.719+0000 7f7faf1c0700 1 -- 172.21.15.152:0/3325425877 >> 172.21.15.152:0/3325425877 conn(0x7f7fa80f09a0 msgr2=0x7f7fa805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:15.721 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.719+0000 7f7faf1c0700 1 -- 172.21.15.152:0/3325425877 shutdown_connections 2023-12-06T04:00:15.721 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:15.719+0000 7f7faf1c0700 1 -- 172.21.15.152:0/3325425877 wait complete. 2023-12-06T04:00:15.722 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:16.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:15 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:14.455290+0000 mgr.smithi132.gokxja (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:17.094 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:17.094 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:17.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:16 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:15.718700+0000 mon.smithi132 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.152:0/3325425877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:18.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:17 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:16.455650+0000 mgr.smithi132.gokxja (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:18.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.177+0000 7fe66399d700 1 -- 172.21.15.152:0/3320720552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe65c0f5410 msgr2=0x7fe65c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:18.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.177+0000 7fe66399d700 1 --2- 172.21.15.152:0/3320720552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe65c0f5410 0x7fe65c0f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fe658009fb0 tx=0x7fe658009b30).stop 2023-12-06T04:00:18.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.177+0000 7fe66399d700 1 -- 172.21.15.152:0/3320720552 shutdown_connections 2023-12-06T04:00:18.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.177+0000 7fe66399d700 1 --2- 172.21.15.152:0/3320720552 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe65c0f5410 0x7fe65c0f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:18.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.177+0000 7fe66399d700 1 -- 172.21.15.152:0/3320720552 >> 172.21.15.152:0/3320720552 conn(0x7fe65c0f0b70 msgr2=0x7fe65c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:18.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.177+0000 7fe66399d700 1 -- 172.21.15.152:0/3320720552 shutdown_connections 2023-12-06T04:00:18.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.177+0000 7fe66399d700 1 -- 172.21.15.152:0/3320720552 wait complete. 2023-12-06T04:00:18.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.178+0000 7fe66399d700 1 Processor -- start 2023-12-06T04:00:18.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.178+0000 7fe66399d700 1 -- start start 2023-12-06T04:00:18.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.178+0000 7fe66399d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe65c0f5410 0x7fe65c1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:18.181 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.178+0000 7fe66399d700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe65c060220 con 0x7fe65c0f5410 2023-12-06T04:00:18.181 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.178+0000 7fe661739700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe65c0f5410 0x7fe65c1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:18.181 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.178+0000 7fe661739700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe65c0f5410 0x7fe65c1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:54600/0 (socket says 172.21.15.152:54600) 2023-12-06T04:00:18.181 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.178+0000 7fe661739700 1 -- 172.21.15.152:0/2795959397 learned_addr learned my addr 172.21.15.152:0/2795959397 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:18.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.178+0000 7fe661739700 1 -- 172.21.15.152:0/2795959397 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe658005040 con 0x7fe65c0f5410 2023-12-06T04:00:18.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.179+0000 7fe661739700 1 --2- 172.21.15.152:0/2795959397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe65c0f5410 0x7fe65c1093f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fe658013510 tx=0x7fe65800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:18.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.179+0000 7fe651ffb700 1 -- 172.21.15.152:0/2795959397 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe658009320 con 0x7fe65c0f5410 2023-12-06T04:00:18.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.179+0000 7fe66399d700 1 -- 172.21.15.152:0/2795959397 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe65c060420 con 0x7fe65c0f5410 2023-12-06T04:00:18.182 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.179+0000 7fe651ffb700 1 -- 172.21.15.152:0/2795959397 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe658013f00 con 0x7fe65c0f5410 2023-12-06T04:00:18.183 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.179+0000 7fe651ffb700 1 -- 172.21.15.152:0/2795959397 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe658014d90 con 0x7fe65c0f5410 2023-12-06T04:00:18.183 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.179+0000 7fe66399d700 1 -- 172.21.15.152:0/2795959397 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe65c109aa0 con 0x7fe65c0f5410 2023-12-06T04:00:18.183 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.180+0000 7fe651ffb700 1 -- 172.21.15.152:0/2795959397 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe6580095f0 con 0x7fe65c0f5410 2023-12-06T04:00:18.183 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.180+0000 7fe66399d700 1 -- 172.21.15.152:0/2795959397 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe65c059100 con 0x7fe65c0f5410 2023-12-06T04:00:18.183 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.180+0000 7fe651ffb700 1 --2- 172.21.15.152:0/2795959397 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe64802ff60 0x7fe648032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:18.183 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.180+0000 7fe651ffb700 1 -- 172.21.15.152:0/2795959397 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe658040fc0 con 0x7fe65c0f5410 2023-12-06T04:00:18.183 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.180+0000 7fe660f38700 1 --2- 172.21.15.152:0/2795959397 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe64802ff60 0x7fe648032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:18.183 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.181+0000 7fe660f38700 1 --2- 172.21.15.152:0/2795959397 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe64802ff60 0x7fe648032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fe64c0096a0 tx=0x7fe64c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:18.186 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.185+0000 7fe651ffb700 1 -- 172.21.15.152:0/2795959397 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe658014420 con 0x7fe65c0f5410 2023-12-06T04:00:18.392 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.390+0000 7fe66399d700 1 -- 172.21.15.152:0/2795959397 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe65c0fae60 con 0x7fe65c0f5410 2023-12-06T04:00:18.392 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.391+0000 7fe651ffb700 1 -- 172.21.15.152:0/2795959397 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe658010230 con 0x7fe65c0f5410 2023-12-06T04:00:18.392 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:18.392 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:18.394 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.392+0000 7fe66399d700 1 -- 172.21.15.152:0/2795959397 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe64802ff60 msgr2=0x7fe648032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:18.394 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.392+0000 7fe66399d700 1 --2- 172.21.15.152:0/2795959397 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe64802ff60 0x7fe648032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fe64c0096a0 tx=0x7fe64c006b40).stop 2023-12-06T04:00:18.394 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.392+0000 7fe66399d700 1 -- 172.21.15.152:0/2795959397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe65c0f5410 msgr2=0x7fe65c1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:18.394 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.392+0000 7fe66399d700 1 --2- 172.21.15.152:0/2795959397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe65c0f5410 0x7fe65c1093f0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fe658013510 tx=0x7fe65800b040).stop 2023-12-06T04:00:18.394 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.392+0000 7fe66399d700 1 -- 172.21.15.152:0/2795959397 shutdown_connections 2023-12-06T04:00:18.394 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.392+0000 7fe66399d700 1 --2- 172.21.15.152:0/2795959397 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fe64802ff60 0x7fe648032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:18.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.392+0000 7fe66399d700 1 --2- 172.21.15.152:0/2795959397 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fe65c0f5410 0x7fe65c1093f0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:18.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.392+0000 7fe66399d700 1 -- 172.21.15.152:0/2795959397 >> 172.21.15.152:0/2795959397 conn(0x7fe65c0f0b70 msgr2=0x7fe65c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:18.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.392+0000 7fe66399d700 1 -- 172.21.15.152:0/2795959397 shutdown_connections 2023-12-06T04:00:18.395 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:18.392+0000 7fe66399d700 1 -- 172.21.15.152:0/2795959397 wait complete. 2023-12-06T04:00:18.395 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:19.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:18 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:18.392078+0000 mon.smithi132 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.152:0/2795959397' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:19.764 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:19.764 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:20.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:19 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:18.456050+0000 mgr.smithi132.gokxja (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:20.966 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.963+0000 7f5917fa1700 1 -- 172.21.15.152:0/3777772159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59100f5410 msgr2=0x7f59100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:20.966 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.963+0000 7f5917fa1700 1 --2- 172.21.15.152:0/3777772159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59100f5410 0x7f59100f5830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f5900009fb0 tx=0x7f5900009b30).stop 2023-12-06T04:00:20.966 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.963+0000 7f5917fa1700 1 -- 172.21.15.152:0/3777772159 shutdown_connections 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.963+0000 7f5917fa1700 1 --2- 172.21.15.152:0/3777772159 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59100f5410 0x7f59100f5830 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.963+0000 7f5917fa1700 1 -- 172.21.15.152:0/3777772159 >> 172.21.15.152:0/3777772159 conn(0x7f59100f0b70 msgr2=0x7f59100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.963+0000 7f5917fa1700 1 -- 172.21.15.152:0/3777772159 shutdown_connections 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.963+0000 7f5917fa1700 1 -- 172.21.15.152:0/3777772159 wait complete. 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.964+0000 7f5917fa1700 1 Processor -- start 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.964+0000 7f5917fa1700 1 -- start start 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.964+0000 7f5917fa1700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59100f5410 0x7f5910109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.964+0000 7f5917fa1700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5910060240 con 0x7f59100f5410 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.964+0000 7f5915d3d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59100f5410 0x7f5910109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.964+0000 7f5915d3d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59100f5410 0x7f5910109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:57512/0 (socket says 172.21.15.152:57512) 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.964+0000 7f5915d3d700 1 -- 172.21.15.152:0/244395093 learned_addr learned my addr 172.21.15.152:0/244395093 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.964+0000 7f5915d3d700 1 -- 172.21.15.152:0/244395093 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5900005040 con 0x7f59100f5410 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.964+0000 7f5915d3d700 1 --2- 172.21.15.152:0/244395093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59100f5410 0x7f5910109430 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f5900014000 tx=0x7f590000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.964+0000 7f59067fc700 1 -- 172.21.15.152:0/244395093 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59000095f0 con 0x7f59100f5410 2023-12-06T04:00:20.967 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.964+0000 7f5917fa1700 1 -- 172.21.15.152:0/244395093 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5910060440 con 0x7f59100f5410 2023-12-06T04:00:20.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.964+0000 7f59067fc700 1 -- 172.21.15.152:0/244395093 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5900007e70 con 0x7f59100f5410 2023-12-06T04:00:20.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.964+0000 7f59067fc700 1 -- 172.21.15.152:0/244395093 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5900014ea0 con 0x7f59100f5410 2023-12-06T04:00:20.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.965+0000 7f5917fa1700 1 -- 172.21.15.152:0/244395093 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5910109ae0 con 0x7f59100f5410 2023-12-06T04:00:20.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.965+0000 7f59067fc700 1 -- 172.21.15.152:0/244395093 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5900025070 con 0x7f59100f5410 2023-12-06T04:00:20.968 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.965+0000 7f5917fa1700 1 -- 172.21.15.152:0/244395093 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5910045ad0 con 0x7f59100f5410 2023-12-06T04:00:20.969 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.966+0000 7f59067fc700 1 --2- 172.21.15.152:0/244395093 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f58fc02ff60 0x7f58fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:20.969 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.966+0000 7f59067fc700 1 -- 172.21.15.152:0/244395093 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5900040d10 con 0x7f59100f5410 2023-12-06T04:00:20.969 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.966+0000 7f591553c700 1 --2- 172.21.15.152:0/244395093 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f58fc02ff60 0x7f58fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:20.969 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.966+0000 7f591553c700 1 --2- 172.21.15.152:0/244395093 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f58fc02ff60 0x7f58fc032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f590c0096a0 tx=0x7f590c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:20.971 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:20.970+0000 7f59067fc700 1 -- 172.21.15.152:0/244395093 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f590000f710 con 0x7f59100f5410 2023-12-06T04:00:21.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T04:00:20.396522+0000 mgr.smithi132.gokxja (mgr.14162) 229 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi132 ... 2023-12-06T04:00:21.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Verifying port 9100 ... 2023-12-06T04:00:21.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Non-zero exit code 1 from systemctl start ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@node-exporter.smithi132 2023-12-06T04:00:21.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: systemctl: stderr Job for ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@node-exporter.smithi132.service failed because a timeout was exceeded. 2023-12-06T04:00:21.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: systemctl: stderr See "systemctl status ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@node-exporter.smithi132.service" and "journalctl -xe" for details. 2023-12-06T04:00:21.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Traceback (most recent call last): 2023-12-06T04:00:21.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-06T04:00:21.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: main() 2023-12-06T04:00:21.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-06T04:00:21.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: r = ctx.func(ctx) 2023-12-06T04:00:21.108 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-06T04:00:21.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: return func(ctx) 2023-12-06T04:00:21.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-06T04:00:21.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: ports=daemon_ports) 2023-12-06T04:00:21.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-06T04:00:21.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-06T04:00:21.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-06T04:00:21.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-06T04:00:21.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-06T04:00:21.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-06T04:00:21.110 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: RuntimeError: Failed command: systemctl start ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@node-exporter.smithi132 2023-12-06T04:00:21.110 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Traceback (most recent call last): 2023-12-06T04:00:21.110 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-12-06T04:00:21.110 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: yield (conn, connr) 2023-12-06T04:00:21.110 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-12-06T04:00:21.110 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: code, '\n'.join(err))) 2023-12-06T04:00:21.110 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi132 ... 2023-12-06T04:00:21.110 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Verifying port 9100 ... 2023-12-06T04:00:21.111 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Non-zero exit code 1 from systemctl start ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@node-exporter.smithi132 2023-12-06T04:00:21.111 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: systemctl: stderr Job for ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@node-exporter.smithi132.service failed because a timeout was exceeded. 2023-12-06T04:00:21.111 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: systemctl: stderr See "systemctl status ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@node-exporter.smithi132.service" and "journalctl -xe" for details. 2023-12-06T04:00:21.111 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: Traceback (most recent call last): 2023-12-06T04:00:21.111 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-12-06T04:00:21.111 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: main() 2023-12-06T04:00:21.111 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-12-06T04:00:21.111 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: r = ctx.func(ctx) 2023-12-06T04:00:21.111 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-12-06T04:00:21.111 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: return func(ctx) 2023-12-06T04:00:21.112 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-12-06T04:00:21.112 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: ports=daemon_ports) 2023-12-06T04:00:21.112 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-12-06T04:00:21.112 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-06T04:00:21.112 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-12-06T04:00:21.112 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-06T04:00:21.112 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: File "/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-12-06T04:00:21.112 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-06T04:00:21.112 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: RuntimeError: Failed command: systemctl start ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@node-exporter.smithi132 2023-12-06T04:00:21.113 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:20 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cephadm 2023-12-06T04:00:20.402336+0000 mgr.smithi132.gokxja (mgr.14162) 230 : cephadm [INF] Deploying daemon prometheus.smithi132 on smithi132 2023-12-06T04:00:21.178 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:21.175+0000 7f5917fa1700 1 -- 172.21.15.152:0/244395093 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59100fae60 con 0x7f59100f5410 2023-12-06T04:00:21.178 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:21.175+0000 7f59067fc700 1 -- 172.21.15.152:0/244395093 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5900012330 con 0x7f59100f5410 2023-12-06T04:00:21.178 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:21.178 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:21.179 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:21.177+0000 7f5917fa1700 1 -- 172.21.15.152:0/244395093 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f58fc02ff60 msgr2=0x7f58fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:21.179 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:21.177+0000 7f5917fa1700 1 --2- 172.21.15.152:0/244395093 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f58fc02ff60 0x7f58fc032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f590c0096a0 tx=0x7f590c006b40).stop 2023-12-06T04:00:21.179 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:21.177+0000 7f5917fa1700 1 -- 172.21.15.152:0/244395093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59100f5410 msgr2=0x7f5910109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:21.179 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:21.177+0000 7f5917fa1700 1 --2- 172.21.15.152:0/244395093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59100f5410 0x7f5910109430 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f5900014000 tx=0x7f590000b040).stop 2023-12-06T04:00:21.179 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:21.177+0000 7f5917fa1700 1 -- 172.21.15.152:0/244395093 shutdown_connections 2023-12-06T04:00:21.179 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:21.177+0000 7f5917fa1700 1 --2- 172.21.15.152:0/244395093 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f58fc02ff60 0x7f58fc032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:21.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:21.177+0000 7f5917fa1700 1 --2- 172.21.15.152:0/244395093 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f59100f5410 0x7f5910109430 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:21.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:21.177+0000 7f5917fa1700 1 -- 172.21.15.152:0/244395093 >> 172.21.15.152:0/244395093 conn(0x7f59100f0b70 msgr2=0x7f59100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:21.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:21.177+0000 7f5917fa1700 1 -- 172.21.15.152:0/244395093 shutdown_connections 2023-12-06T04:00:21.180 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:21.177+0000 7f5917fa1700 1 -- 172.21.15.152:0/244395093 wait complete. 2023-12-06T04:00:21.180 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:21.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:20.456374+0000 mgr.smithi132.gokxja (mgr.14162) 231 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:21.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:21.176830+0000 mon.smithi132 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.152:0/244395093' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:22.569 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:22.569 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:23.690 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.689+0000 7fd014ce3700 1 -- 172.21.15.152:0/2147397712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010060280 msgr2=0x7fd0100f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:23.691 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.689+0000 7fd014ce3700 1 --2- 172.21.15.152:0/2147397712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010060280 0x7fd0100f90f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fd004009fb0 tx=0x7fd004009b30).stop 2023-12-06T04:00:23.691 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.689+0000 7fd014ce3700 1 -- 172.21.15.152:0/2147397712 shutdown_connections 2023-12-06T04:00:23.691 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.689+0000 7fd014ce3700 1 --2- 172.21.15.152:0/2147397712 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010060280 0x7fd0100f90f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:23.691 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.689+0000 7fd014ce3700 1 -- 172.21.15.152:0/2147397712 >> 172.21.15.152:0/2147397712 conn(0x7fd0100f0b70 msgr2=0x7fd0100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:23.691 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.689+0000 7fd014ce3700 1 -- 172.21.15.152:0/2147397712 shutdown_connections 2023-12-06T04:00:23.691 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.689+0000 7fd014ce3700 1 -- 172.21.15.152:0/2147397712 wait complete. 2023-12-06T04:00:23.691 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.689+0000 7fd014ce3700 1 Processor -- start 2023-12-06T04:00:23.691 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.689+0000 7fd014ce3700 1 -- start start 2023-12-06T04:00:23.691 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.689+0000 7fd014ce3700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010060280 0x7fd0100f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:23.691 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.689+0000 7fd014ce3700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0100f4ff0 con 0x7fd010060280 2023-12-06T04:00:23.691 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.690+0000 7fd00e59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010060280 0x7fd0100f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:23.692 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.690+0000 7fd00e59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010060280 0x7fd0100f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:57526/0 (socket says 172.21.15.152:57526) 2023-12-06T04:00:23.692 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.690+0000 7fd00e59c700 1 -- 172.21.15.152:0/171308563 learned_addr learned my addr 172.21.15.152:0/171308563 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:23.692 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.690+0000 7fd00e59c700 1 -- 172.21.15.152:0/171308563 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd004005040 con 0x7fd010060280 2023-12-06T04:00:23.692 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.690+0000 7fd00e59c700 1 --2- 172.21.15.152:0/171308563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010060280 0x7fd0100f6b70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fd004013510 tx=0x7fd00400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:23.692 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.690+0000 7fcffeffd700 1 -- 172.21.15.152:0/171308563 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd004009320 con 0x7fd010060280 2023-12-06T04:00:23.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.690+0000 7fd014ce3700 1 -- 172.21.15.152:0/171308563 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0100f5190 con 0x7fd010060280 2023-12-06T04:00:23.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.690+0000 7fcffeffd700 1 -- 172.21.15.152:0/171308563 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd004013f00 con 0x7fd010060280 2023-12-06T04:00:23.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.690+0000 7fcffeffd700 1 -- 172.21.15.152:0/171308563 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd004014dc0 con 0x7fd010060280 2023-12-06T04:00:23.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.690+0000 7fd014ce3700 1 -- 172.21.15.152:0/171308563 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0100f5620 con 0x7fd010060280 2023-12-06T04:00:23.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.691+0000 7fcffeffd700 1 -- 172.21.15.152:0/171308563 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0040095f0 con 0x7fd010060280 2023-12-06T04:00:23.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.691+0000 7fd014ce3700 1 -- 172.21.15.152:0/171308563 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd010059100 con 0x7fd010060280 2023-12-06T04:00:23.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.691+0000 7fcffeffd700 1 --2- 172.21.15.152:0/171308563 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcff802ffb0 0x7fcff8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:23.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.691+0000 7fcffeffd700 1 -- 172.21.15.152:0/171308563 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd004040fe0 con 0x7fd010060280 2023-12-06T04:00:23.693 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.691+0000 7fd00dd9b700 1 --2- 172.21.15.152:0/171308563 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcff802ffb0 0x7fcff8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:23.694 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.692+0000 7fd00dd9b700 1 --2- 172.21.15.152:0/171308563 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcff802ffb0 0x7fcff8032470 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fd0000096a0 tx=0x7fd000006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:23.696 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.696+0000 7fcffeffd700 1 -- 172.21.15.152:0/171308563 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd00406d2b0 con 0x7fd010060280 2023-12-06T04:00:23.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.904+0000 7fd014ce3700 1 -- 172.21.15.152:0/171308563 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0100fbf50 con 0x7fd010060280 2023-12-06T04:00:23.906 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.905+0000 7fcffeffd700 1 -- 172.21.15.152:0/171308563 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd004014420 con 0x7fd010060280 2023-12-06T04:00:23.906 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:23.906 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:23.908 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.907+0000 7fd014ce3700 1 -- 172.21.15.152:0/171308563 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcff802ffb0 msgr2=0x7fcff8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:23.908 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.907+0000 7fd014ce3700 1 --2- 172.21.15.152:0/171308563 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcff802ffb0 0x7fcff8032470 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fd0000096a0 tx=0x7fd000006b40).stop 2023-12-06T04:00:23.908 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.907+0000 7fd014ce3700 1 -- 172.21.15.152:0/171308563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010060280 msgr2=0x7fd0100f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:23.908 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.907+0000 7fd014ce3700 1 --2- 172.21.15.152:0/171308563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010060280 0x7fd0100f6b70 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fd004013510 tx=0x7fd00400b040).stop 2023-12-06T04:00:23.908 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.907+0000 7fd014ce3700 1 -- 172.21.15.152:0/171308563 shutdown_connections 2023-12-06T04:00:23.908 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.907+0000 7fd014ce3700 1 --2- 172.21.15.152:0/171308563 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fcff802ffb0 0x7fcff8032470 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:23.909 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.907+0000 7fd014ce3700 1 --2- 172.21.15.152:0/171308563 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd010060280 0x7fd0100f6b70 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:23.909 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.907+0000 7fd014ce3700 1 -- 172.21.15.152:0/171308563 >> 172.21.15.152:0/171308563 conn(0x7fd0100f0b70 msgr2=0x7fd0100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:23.909 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.907+0000 7fd014ce3700 1 -- 172.21.15.152:0/171308563 shutdown_connections 2023-12-06T04:00:23.909 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:23.907+0000 7fd014ce3700 1 -- 172.21.15.152:0/171308563 wait complete. 2023-12-06T04:00:23.909 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:24.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:22.456709+0000 mgr.smithi132.gokxja (mgr.14162) 232 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:25.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:24 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:23.905818+0000 mon.smithi132 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.152:0/171308563' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:25.275 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:25.276 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:26.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:24.457028+0000 mgr.smithi132.gokxja (mgr.14162) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:26.386 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.385+0000 7ffabbb86700 1 -- 172.21.15.152:0/4129355503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffab40f6510 msgr2=0x7ffab40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:26.386 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.385+0000 7ffabbb86700 1 --2- 172.21.15.152:0/4129355503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffab40f6510 0x7ffab40f6930 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7ffab0009fb0 tx=0x7ffab0009b30).stop 2023-12-06T04:00:26.387 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.385+0000 7ffabbb86700 1 -- 172.21.15.152:0/4129355503 shutdown_connections 2023-12-06T04:00:26.387 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.385+0000 7ffabbb86700 1 --2- 172.21.15.152:0/4129355503 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffab40f6510 0x7ffab40f6930 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:26.387 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.385+0000 7ffabbb86700 1 -- 172.21.15.152:0/4129355503 >> 172.21.15.152:0/4129355503 conn(0x7ffab40f1c70 msgr2=0x7ffab40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:26.387 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.385+0000 7ffabbb86700 1 -- 172.21.15.152:0/4129355503 shutdown_connections 2023-12-06T04:00:26.387 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.385+0000 7ffabbb86700 1 -- 172.21.15.152:0/4129355503 wait complete. 2023-12-06T04:00:26.387 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.386+0000 7ffabbb86700 1 Processor -- start 2023-12-06T04:00:26.387 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.386+0000 7ffabbb86700 1 -- start start 2023-12-06T04:00:26.387 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.386+0000 7ffabbb86700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffab40f6510 0x7ffab4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:26.387 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.386+0000 7ffabbb86700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffab4106c20 con 0x7ffab40f6510 2023-12-06T04:00:26.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.386+0000 7ffab9922700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffab40f6510 0x7ffab4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:26.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.386+0000 7ffab9922700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffab40f6510 0x7ffab4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:57534/0 (socket says 172.21.15.152:57534) 2023-12-06T04:00:26.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.386+0000 7ffab9922700 1 -- 172.21.15.152:0/552560266 learned_addr learned my addr 172.21.15.152:0/552560266 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:26.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.386+0000 7ffab9922700 1 -- 172.21.15.152:0/552560266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffab0005040 con 0x7ffab40f6510 2023-12-06T04:00:26.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.386+0000 7ffab9922700 1 --2- 172.21.15.152:0/552560266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffab40f6510 0x7ffab4106730 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7ffab0013510 tx=0x7ffab000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:26.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.387+0000 7ffaaa7fc700 1 -- 172.21.15.152:0/552560266 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffab0009320 con 0x7ffab40f6510 2023-12-06T04:00:26.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.387+0000 7ffabbb86700 1 -- 172.21.15.152:0/552560266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffab4106e20 con 0x7ffab40f6510 2023-12-06T04:00:26.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.387+0000 7ffaaa7fc700 1 -- 172.21.15.152:0/552560266 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffab0013f00 con 0x7ffab40f6510 2023-12-06T04:00:26.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.387+0000 7ffaaa7fc700 1 -- 172.21.15.152:0/552560266 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffab0014d90 con 0x7ffab40f6510 2023-12-06T04:00:26.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.387+0000 7ffabbb86700 1 -- 172.21.15.152:0/552560266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffab41072b0 con 0x7ffab40f6510 2023-12-06T04:00:26.389 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.388+0000 7ffaaa7fc700 1 -- 172.21.15.152:0/552560266 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffab00095f0 con 0x7ffab40f6510 2023-12-06T04:00:26.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.388+0000 7ffabbb86700 1 -- 172.21.15.152:0/552560266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffab4045ad0 con 0x7ffab40f6510 2023-12-06T04:00:26.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.388+0000 7ffaaa7fc700 1 --2- 172.21.15.152:0/552560266 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffaa002ff60 0x7ffaa0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:26.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.388+0000 7ffaaa7fc700 1 -- 172.21.15.152:0/552560266 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffab00410c0 con 0x7ffab40f6510 2023-12-06T04:00:26.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.388+0000 7ffab9121700 1 --2- 172.21.15.152:0/552560266 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffaa002ff60 0x7ffaa0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:26.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.388+0000 7ffab9121700 1 --2- 172.21.15.152:0/552560266 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffaa002ff60 0x7ffaa0032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ffaa40096a0 tx=0x7ffaa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:26.393 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.392+0000 7ffaaa7fc700 1 -- 172.21.15.152:0/552560266 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffab0014420 con 0x7ffab40f6510 2023-12-06T04:00:26.600 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.599+0000 7ffabbb86700 1 -- 172.21.15.152:0/552560266 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffab40fbf60 con 0x7ffab40f6510 2023-12-06T04:00:26.601 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.600+0000 7ffaaa7fc700 1 -- 172.21.15.152:0/552560266 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffab0010230 con 0x7ffab40f6510 2023-12-06T04:00:26.601 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:26.601 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:26.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.602+0000 7ffabbb86700 1 -- 172.21.15.152:0/552560266 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffaa002ff60 msgr2=0x7ffaa0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:26.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.602+0000 7ffabbb86700 1 --2- 172.21.15.152:0/552560266 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffaa002ff60 0x7ffaa0032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7ffaa40096a0 tx=0x7ffaa4006b40).stop 2023-12-06T04:00:26.602 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.602+0000 7ffabbb86700 1 -- 172.21.15.152:0/552560266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffab40f6510 msgr2=0x7ffab4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:26.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.602+0000 7ffabbb86700 1 --2- 172.21.15.152:0/552560266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffab40f6510 0x7ffab4106730 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7ffab0013510 tx=0x7ffab000b040).stop 2023-12-06T04:00:26.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.602+0000 7ffabbb86700 1 -- 172.21.15.152:0/552560266 shutdown_connections 2023-12-06T04:00:26.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.602+0000 7ffabbb86700 1 --2- 172.21.15.152:0/552560266 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffaa002ff60 0x7ffaa0032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:26.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.602+0000 7ffabbb86700 1 --2- 172.21.15.152:0/552560266 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffab40f6510 0x7ffab4106730 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:26.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.602+0000 7ffabbb86700 1 -- 172.21.15.152:0/552560266 >> 172.21.15.152:0/552560266 conn(0x7ffab40f1c70 msgr2=0x7ffab40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:26.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.602+0000 7ffabbb86700 1 -- 172.21.15.152:0/552560266 shutdown_connections 2023-12-06T04:00:26.603 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:26.602+0000 7ffabbb86700 1 -- 172.21.15.152:0/552560266 wait complete. 2023-12-06T04:00:26.603 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:27.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:26.457389+0000 mgr.smithi132.gokxja (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:27.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:26.600659+0000 mon.smithi132 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.152:0/552560266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:27.969 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:27.970 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:29.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.098+0000 7fd87545e700 1 -- 172.21.15.152:0/3093173710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8700f6510 msgr2=0x7fd8700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:29.099 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.098+0000 7fd87545e700 1 --2- 172.21.15.152:0/3093173710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8700f6510 0x7fd8700f6930 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fd864009fb0 tx=0x7fd864009b30).stop 2023-12-06T04:00:29.100 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.098+0000 7fd87545e700 1 -- 172.21.15.152:0/3093173710 shutdown_connections 2023-12-06T04:00:29.100 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.098+0000 7fd87545e700 1 --2- 172.21.15.152:0/3093173710 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8700f6510 0x7fd8700f6930 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:29.100 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.098+0000 7fd87545e700 1 -- 172.21.15.152:0/3093173710 >> 172.21.15.152:0/3093173710 conn(0x7fd8700f1c70 msgr2=0x7fd8700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:29.100 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.098+0000 7fd87545e700 1 -- 172.21.15.152:0/3093173710 shutdown_connections 2023-12-06T04:00:29.100 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.098+0000 7fd87545e700 1 -- 172.21.15.152:0/3093173710 wait complete. 2023-12-06T04:00:29.101 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.099+0000 7fd87545e700 1 Processor -- start 2023-12-06T04:00:29.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.099+0000 7fd87545e700 1 -- start start 2023-12-06T04:00:29.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.099+0000 7fd87545e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8700f6510 0x7fd87006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:29.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.099+0000 7fd87545e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd870069e20 con 0x7fd8700f6510 2023-12-06T04:00:29.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.099+0000 7fd86effd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8700f6510 0x7fd87006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:29.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.099+0000 7fd86effd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8700f6510 0x7fd87006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:57566/0 (socket says 172.21.15.152:57566) 2023-12-06T04:00:29.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.099+0000 7fd86effd700 1 -- 172.21.15.152:0/3851318908 learned_addr learned my addr 172.21.15.152:0/3851318908 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:29.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.100+0000 7fd86effd700 1 -- 172.21.15.152:0/3851318908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd864005040 con 0x7fd8700f6510 2023-12-06T04:00:29.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.100+0000 7fd86effd700 1 --2- 172.21.15.152:0/3851318908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8700f6510 0x7fd87006b9a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fd864013510 tx=0x7fd86400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:29.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.100+0000 7fd8577fe700 1 -- 172.21.15.152:0/3851318908 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd864009320 con 0x7fd8700f6510 2023-12-06T04:00:29.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.100+0000 7fd87545e700 1 -- 172.21.15.152:0/3851318908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd870069fc0 con 0x7fd8700f6510 2023-12-06T04:00:29.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.100+0000 7fd8577fe700 1 -- 172.21.15.152:0/3851318908 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd864013f00 con 0x7fd8700f6510 2023-12-06T04:00:29.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.100+0000 7fd8577fe700 1 -- 172.21.15.152:0/3851318908 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd864014d90 con 0x7fd8700f6510 2023-12-06T04:00:29.102 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.100+0000 7fd87545e700 1 -- 172.21.15.152:0/3851318908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd87006a450 con 0x7fd8700f6510 2023-12-06T04:00:29.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.101+0000 7fd8577fe700 1 -- 172.21.15.152:0/3851318908 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8640263a0 con 0x7fd8700f6510 2023-12-06T04:00:29.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.101+0000 7fd8577fe700 1 --2- 172.21.15.152:0/3851318908 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd85802ff60 0x7fd858032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:29.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.101+0000 7fd8577fe700 1 -- 172.21.15.152:0/3851318908 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd8640410f0 con 0x7fd8700f6510 2023-12-06T04:00:29.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.101+0000 7fd87545e700 1 -- 172.21.15.152:0/3851318908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd870059100 con 0x7fd8700f6510 2023-12-06T04:00:29.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.101+0000 7fd86e7fc700 1 --2- 172.21.15.152:0/3851318908 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd85802ff60 0x7fd858032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:29.103 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.102+0000 7fd86e7fc700 1 --2- 172.21.15.152:0/3851318908 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd85802ff60 0x7fd858032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fd8600096a0 tx=0x7fd860006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:29.107 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.106+0000 7fd8577fe700 1 -- 172.21.15.152:0/3851318908 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd864012920 con 0x7fd8700f6510 2023-12-06T04:00:29.314 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.313+0000 7fd87545e700 1 -- 172.21.15.152:0/3851318908 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd87006ab50 con 0x7fd8700f6510 2023-12-06T04:00:29.315 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.314+0000 7fd8577fe700 1 -- 172.21.15.152:0/3851318908 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd86402b020 con 0x7fd8700f6510 2023-12-06T04:00:29.315 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:29.315 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:29.316 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.315+0000 7fd87545e700 1 -- 172.21.15.152:0/3851318908 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd85802ff60 msgr2=0x7fd858032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:29.316 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.315+0000 7fd87545e700 1 --2- 172.21.15.152:0/3851318908 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd85802ff60 0x7fd858032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fd8600096a0 tx=0x7fd860006b40).stop 2023-12-06T04:00:29.316 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.315+0000 7fd87545e700 1 -- 172.21.15.152:0/3851318908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8700f6510 msgr2=0x7fd87006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:29.316 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.315+0000 7fd87545e700 1 --2- 172.21.15.152:0/3851318908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8700f6510 0x7fd87006b9a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7fd864013510 tx=0x7fd86400b040).stop 2023-12-06T04:00:29.316 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.315+0000 7fd87545e700 1 -- 172.21.15.152:0/3851318908 shutdown_connections 2023-12-06T04:00:29.317 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.316+0000 7fd87545e700 1 --2- 172.21.15.152:0/3851318908 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fd85802ff60 0x7fd858032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:29.317 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.316+0000 7fd87545e700 1 --2- 172.21.15.152:0/3851318908 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fd8700f6510 0x7fd87006b9a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:29.317 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.316+0000 7fd87545e700 1 -- 172.21.15.152:0/3851318908 >> 172.21.15.152:0/3851318908 conn(0x7fd8700f1c70 msgr2=0x7fd8700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:29.317 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.316+0000 7fd87545e700 1 -- 172.21.15.152:0/3851318908 shutdown_connections 2023-12-06T04:00:29.317 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:29.316+0000 7fd87545e700 1 -- 172.21.15.152:0/3851318908 wait complete. 2023-12-06T04:00:29.317 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:29.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:28.457660+0000 mgr.smithi132.gokxja (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:29.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:29.314638+0000 mon.smithi132 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.152:0/3851318908' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:30.680 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:30.681 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:31.853 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.851+0000 7f46ee76f700 1 -- 172.21.15.152:0/1951280342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46e8060280 msgr2=0x7f46e80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:31.853 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.851+0000 7f46ee76f700 1 --2- 172.21.15.152:0/1951280342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46e8060280 0x7f46e80f90f0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f46d8009fb0 tx=0x7f46d8009b30).stop 2023-12-06T04:00:31.853 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.851+0000 7f46ee76f700 1 -- 172.21.15.152:0/1951280342 shutdown_connections 2023-12-06T04:00:31.853 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.851+0000 7f46ee76f700 1 --2- 172.21.15.152:0/1951280342 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46e8060280 0x7f46e80f90f0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:31.853 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.851+0000 7f46ee76f700 1 -- 172.21.15.152:0/1951280342 >> 172.21.15.152:0/1951280342 conn(0x7f46e80f0b70 msgr2=0x7f46e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:31.853 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.852+0000 7f46ee76f700 1 -- 172.21.15.152:0/1951280342 shutdown_connections 2023-12-06T04:00:31.853 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.852+0000 7f46ee76f700 1 -- 172.21.15.152:0/1951280342 wait complete. 2023-12-06T04:00:31.854 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.852+0000 7f46ee76f700 1 Processor -- start 2023-12-06T04:00:31.854 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.852+0000 7f46ee76f700 1 -- start start 2023-12-06T04:00:31.854 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.852+0000 7f46ee76f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46e8060280 0x7f46e81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:31.855 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.852+0000 7f46ee76f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46e8106bd0 con 0x7f46e8060280 2023-12-06T04:00:31.855 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.853+0000 7f46e7fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46e8060280 0x7f46e81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:31.855 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.853+0000 7f46e7fff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46e8060280 0x7f46e81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:47050/0 (socket says 172.21.15.152:47050) 2023-12-06T04:00:31.855 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.853+0000 7f46e7fff700 1 -- 172.21.15.152:0/4243669705 learned_addr learned my addr 172.21.15.152:0/4243669705 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:31.855 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.853+0000 7f46e7fff700 1 -- 172.21.15.152:0/4243669705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d8005040 con 0x7f46e8060280 2023-12-06T04:00:31.855 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.853+0000 7f46e7fff700 1 --2- 172.21.15.152:0/4243669705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46e8060280 0x7f46e81066e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f46d8009830 tx=0x7f46d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:31.855 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.853+0000 7f46e4ff9700 1 -- 172.21.15.152:0/4243669705 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46d8009320 con 0x7f46e8060280 2023-12-06T04:00:31.855 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.853+0000 7f46ee76f700 1 -- 172.21.15.152:0/4243669705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46e8106dd0 con 0x7f46e8060280 2023-12-06T04:00:31.855 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.853+0000 7f46e4ff9700 1 -- 172.21.15.152:0/4243669705 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46d80136c0 con 0x7f46e8060280 2023-12-06T04:00:31.855 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.853+0000 7f46e4ff9700 1 -- 172.21.15.152:0/4243669705 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46d801ee90 con 0x7f46e8060280 2023-12-06T04:00:31.855 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.853+0000 7f46ee76f700 1 -- 172.21.15.152:0/4243669705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46e8107260 con 0x7f46e8060280 2023-12-06T04:00:31.856 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.854+0000 7f46e4ff9700 1 -- 172.21.15.152:0/4243669705 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f46d801e420 con 0x7f46e8060280 2023-12-06T04:00:31.856 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.854+0000 7f46ee76f700 1 -- 172.21.15.152:0/4243669705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46e8045ad0 con 0x7f46e8060280 2023-12-06T04:00:31.856 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.854+0000 7f46e4ff9700 1 --2- 172.21.15.152:0/4243669705 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f46d002ff10 0x7f46d00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:31.856 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.854+0000 7f46e4ff9700 1 -- 172.21.15.152:0/4243669705 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f46d80410d0 con 0x7f46e8060280 2023-12-06T04:00:31.856 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.854+0000 7f46e77fe700 1 --2- 172.21.15.152:0/4243669705 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f46d002ff10 0x7f46d00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:31.856 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.855+0000 7f46e77fe700 1 --2- 172.21.15.152:0/4243669705 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f46d002ff10 0x7f46d00323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f46dc0096a0 tx=0x7f46dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:31.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:31 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:30.457918+0000 mgr.smithi132.gokxja (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:31.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:31.859+0000 7f46e4ff9700 1 -- 172.21.15.152:0/4243669705 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46d80095f0 con 0x7f46e8060280 2023-12-06T04:00:32.064 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:32.064+0000 7f46ee76f700 1 -- 172.21.15.152:0/4243669705 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f46e8108110 con 0x7f46e8060280 2023-12-06T04:00:32.065 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:32.065+0000 7f46e4ff9700 1 -- 172.21.15.152:0/4243669705 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f46d802c2d0 con 0x7f46e8060280 2023-12-06T04:00:32.065 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:32.065 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:32.066 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:32.066+0000 7f46ee76f700 1 -- 172.21.15.152:0/4243669705 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f46d002ff10 msgr2=0x7f46d00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:32.069 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:32.066+0000 7f46ee76f700 1 --2- 172.21.15.152:0/4243669705 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f46d002ff10 0x7f46d00323d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f46dc0096a0 tx=0x7f46dc006b40).stop 2023-12-06T04:00:32.069 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:32.066+0000 7f46ee76f700 1 -- 172.21.15.152:0/4243669705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46e8060280 msgr2=0x7f46e81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:32.069 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:32.066+0000 7f46ee76f700 1 --2- 172.21.15.152:0/4243669705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46e8060280 0x7f46e81066e0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f46d8009830 tx=0x7f46d800b040).stop 2023-12-06T04:00:32.070 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:32.066+0000 7f46ee76f700 1 -- 172.21.15.152:0/4243669705 shutdown_connections 2023-12-06T04:00:32.070 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:32.066+0000 7f46ee76f700 1 --2- 172.21.15.152:0/4243669705 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f46d002ff10 0x7f46d00323d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:32.070 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:32.066+0000 7f46ee76f700 1 --2- 172.21.15.152:0/4243669705 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f46e8060280 0x7f46e81066e0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:32.070 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:32.066+0000 7f46ee76f700 1 -- 172.21.15.152:0/4243669705 >> 172.21.15.152:0/4243669705 conn(0x7f46e80f0b70 msgr2=0x7f46e80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:32.070 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:32.066+0000 7f46ee76f700 1 -- 172.21.15.152:0/4243669705 shutdown_connections 2023-12-06T04:00:32.070 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:32.066+0000 7f46ee76f700 1 -- 172.21.15.152:0/4243669705 wait complete. 2023-12-06T04:00:32.070 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:32.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:32 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:32.065058+0000 mon.smithi132 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.152:0/4243669705' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:33.431 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:33.432 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:33.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:33 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:32.458237+0000 mgr.smithi132.gokxja (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:34.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.484+0000 7f826a838700 1 -- 172.21.15.152:0/2107222051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82640f51a0 msgr2=0x7f82640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:34.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.485+0000 7f826a838700 1 --2- 172.21.15.152:0/2107222051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82640f51a0 0x7f82640f55c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f8260009fb0 tx=0x7f8260009b30).stop 2023-12-06T04:00:34.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.485+0000 7f826a838700 1 -- 172.21.15.152:0/2107222051 shutdown_connections 2023-12-06T04:00:34.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.485+0000 7f826a838700 1 --2- 172.21.15.152:0/2107222051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82640f51a0 0x7f82640f55c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:34.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.485+0000 7f826a838700 1 -- 172.21.15.152:0/2107222051 >> 172.21.15.152:0/2107222051 conn(0x7f82640f09a0 msgr2=0x7f82640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:34.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.485+0000 7f826a838700 1 -- 172.21.15.152:0/2107222051 shutdown_connections 2023-12-06T04:00:34.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.485+0000 7f826a838700 1 -- 172.21.15.152:0/2107222051 wait complete. 2023-12-06T04:00:34.487 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.485+0000 7f826a838700 1 Processor -- start 2023-12-06T04:00:34.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.485+0000 7f826a838700 1 -- start start 2023-12-06T04:00:34.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.485+0000 7f826a838700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82640f51a0 0x7f82641091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:34.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.485+0000 7f826a838700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8264060240 con 0x7f82640f51a0 2023-12-06T04:00:34.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.486+0000 7f8269836700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82640f51a0 0x7f82641091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:34.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.486+0000 7f8269836700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82640f51a0 0x7f82641091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:47070/0 (socket says 172.21.15.152:47070) 2023-12-06T04:00:34.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.486+0000 7f8269836700 1 -- 172.21.15.152:0/962617946 learned_addr learned my addr 172.21.15.152:0/962617946 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:34.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.486+0000 7f8269836700 1 -- 172.21.15.152:0/962617946 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8260005040 con 0x7f82640f51a0 2023-12-06T04:00:34.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.486+0000 7f8269836700 1 --2- 172.21.15.152:0/962617946 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82640f51a0 0x7f82641091b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f8260013510 tx=0x7f826000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:34.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.486+0000 7f825a7fc700 1 -- 172.21.15.152:0/962617946 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8260009320 con 0x7f82640f51a0 2023-12-06T04:00:34.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.486+0000 7f826a838700 1 -- 172.21.15.152:0/962617946 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8264060440 con 0x7f82640f51a0 2023-12-06T04:00:34.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.486+0000 7f825a7fc700 1 -- 172.21.15.152:0/962617946 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8260013f00 con 0x7f82640f51a0 2023-12-06T04:00:34.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.486+0000 7f825a7fc700 1 -- 172.21.15.152:0/962617946 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8260014dc0 con 0x7f82640f51a0 2023-12-06T04:00:34.488 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.486+0000 7f826a838700 1 -- 172.21.15.152:0/962617946 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8264109860 con 0x7f82640f51a0 2023-12-06T04:00:34.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.487+0000 7f825a7fc700 1 -- 172.21.15.152:0/962617946 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f82600095f0 con 0x7f82640f51a0 2023-12-06T04:00:34.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.487+0000 7f825a7fc700 1 --2- 172.21.15.152:0/962617946 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f825002ff60 0x7f8250032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:34.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.487+0000 7f826a838700 1 -- 172.21.15.152:0/962617946 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8264045ad0 con 0x7f82640f51a0 2023-12-06T04:00:34.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.487+0000 7f825a7fc700 1 -- 172.21.15.152:0/962617946 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f82600410c0 con 0x7f82640f51a0 2023-12-06T04:00:34.489 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.487+0000 7f8269035700 1 --2- 172.21.15.152:0/962617946 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f825002ff60 0x7f8250032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:34.490 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.488+0000 7f8269035700 1 --2- 172.21.15.152:0/962617946 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f825002ff60 0x7f8250032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f82540096a0 tx=0x7f8254006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:34.493 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.492+0000 7f825a7fc700 1 -- 172.21.15.152:0/962617946 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f826006d2b0 con 0x7f82640f51a0 2023-12-06T04:00:34.699 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.698+0000 7f826a838700 1 -- 172.21.15.152:0/962617946 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f826410a080 con 0x7f82640f51a0 2023-12-06T04:00:34.700 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.699+0000 7f825a7fc700 1 -- 172.21.15.152:0/962617946 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8260010230 con 0x7f82640f51a0 2023-12-06T04:00:34.700 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:34.700 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:34.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.701+0000 7f826a838700 1 -- 172.21.15.152:0/962617946 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f825002ff60 msgr2=0x7f8250032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:34.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.701+0000 7f826a838700 1 --2- 172.21.15.152:0/962617946 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f825002ff60 0x7f8250032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f82540096a0 tx=0x7f8254006b40).stop 2023-12-06T04:00:34.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.701+0000 7f826a838700 1 -- 172.21.15.152:0/962617946 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82640f51a0 msgr2=0x7f82641091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:34.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.701+0000 7f826a838700 1 --2- 172.21.15.152:0/962617946 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82640f51a0 0x7f82641091b0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f8260013510 tx=0x7f826000b040).stop 2023-12-06T04:00:34.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.701+0000 7f826a838700 1 -- 172.21.15.152:0/962617946 shutdown_connections 2023-12-06T04:00:34.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.701+0000 7f826a838700 1 --2- 172.21.15.152:0/962617946 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f825002ff60 0x7f8250032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:34.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.701+0000 7f826a838700 1 --2- 172.21.15.152:0/962617946 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f82640f51a0 0x7f82641091b0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:34.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.701+0000 7f826a838700 1 -- 172.21.15.152:0/962617946 >> 172.21.15.152:0/962617946 conn(0x7f82640f09a0 msgr2=0x7f82640f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:34.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.701+0000 7f826a838700 1 -- 172.21.15.152:0/962617946 shutdown_connections 2023-12-06T04:00:34.702 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:34.701+0000 7f826a838700 1 -- 172.21.15.152:0/962617946 wait complete. 2023-12-06T04:00:34.703 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:35.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:35 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:34.458612+0000 mgr.smithi132.gokxja (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:35.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:35 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:34.699742+0000 mon.smithi132 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.152:0/962617946' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:36.078 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:36.078 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:37.171 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.171+0000 7fc91dbc2700 1 -- 172.21.15.152:0/2986353019 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9180f62a0 msgr2=0x7fc9180f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:37.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.171+0000 7fc91dbc2700 1 --2- 172.21.15.152:0/2986353019 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9180f62a0 0x7fc9180f66c0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fc90c009fb0 tx=0x7fc90c009b30).stop 2023-12-06T04:00:37.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.171+0000 7fc91dbc2700 1 -- 172.21.15.152:0/2986353019 shutdown_connections 2023-12-06T04:00:37.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.171+0000 7fc91dbc2700 1 --2- 172.21.15.152:0/2986353019 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9180f62a0 0x7fc9180f66c0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:37.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.171+0000 7fc91dbc2700 1 -- 172.21.15.152:0/2986353019 >> 172.21.15.152:0/2986353019 conn(0x7fc9180f1aa0 msgr2=0x7fc9180f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:37.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.171+0000 7fc91dbc2700 1 -- 172.21.15.152:0/2986353019 shutdown_connections 2023-12-06T04:00:37.172 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.171+0000 7fc91dbc2700 1 -- 172.21.15.152:0/2986353019 wait complete. 2023-12-06T04:00:37.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.171+0000 7fc91dbc2700 1 Processor -- start 2023-12-06T04:00:37.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.171+0000 7fc91dbc2700 1 -- start start 2023-12-06T04:00:37.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.172+0000 7fc91dbc2700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9180f62a0 0x7fc9181064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:37.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.172+0000 7fc91dbc2700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9181069a0 con 0x7fc9180f62a0 2023-12-06T04:00:37.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.172+0000 7fc91cbc0700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9180f62a0 0x7fc9181064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:37.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.172+0000 7fc91cbc0700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9180f62a0 0x7fc9181064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:47078/0 (socket says 172.21.15.152:47078) 2023-12-06T04:00:37.173 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.172+0000 7fc91cbc0700 1 -- 172.21.15.152:0/1285042493 learned_addr learned my addr 172.21.15.152:0/1285042493 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:37.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.173+0000 7fc91cbc0700 1 -- 172.21.15.152:0/1285042493 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc90c005040 con 0x7fc9180f62a0 2023-12-06T04:00:37.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.173+0000 7fc91cbc0700 1 --2- 172.21.15.152:0/1285042493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9180f62a0 0x7fc9181064b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fc90c013510 tx=0x7fc90c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:37.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.173+0000 7fc9157fa700 1 -- 172.21.15.152:0/1285042493 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc90c009320 con 0x7fc9180f62a0 2023-12-06T04:00:37.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.173+0000 7fc91dbc2700 1 -- 172.21.15.152:0/1285042493 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc918106ba0 con 0x7fc9180f62a0 2023-12-06T04:00:37.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.173+0000 7fc9157fa700 1 -- 172.21.15.152:0/1285042493 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc90c013f00 con 0x7fc9180f62a0 2023-12-06T04:00:37.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.173+0000 7fc9157fa700 1 -- 172.21.15.152:0/1285042493 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc90c014dc0 con 0x7fc9180f62a0 2023-12-06T04:00:37.174 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.173+0000 7fc91dbc2700 1 -- 172.21.15.152:0/1285042493 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc918107030 con 0x7fc9180f62a0 2023-12-06T04:00:37.175 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.174+0000 7fc9157fa700 1 -- 172.21.15.152:0/1285042493 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc90c0095f0 con 0x7fc9180f62a0 2023-12-06T04:00:37.175 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.174+0000 7fc91dbc2700 1 -- 172.21.15.152:0/1285042493 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc918045ad0 con 0x7fc9180f62a0 2023-12-06T04:00:37.175 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.174+0000 7fc9157fa700 1 --2- 172.21.15.152:0/1285042493 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc90802ff60 0x7fc908032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:37.176 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.174+0000 7fc9157fa700 1 -- 172.21.15.152:0/1285042493 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc90c041020 con 0x7fc9180f62a0 2023-12-06T04:00:37.176 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.174+0000 7fc917fff700 1 --2- 172.21.15.152:0/1285042493 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc90802ff60 0x7fc908032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:37.176 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.175+0000 7fc917fff700 1 --2- 172.21.15.152:0/1285042493 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc90802ff60 0x7fc908032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fc9040096a0 tx=0x7fc904006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:37.179 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.179+0000 7fc9157fa700 1 -- 172.21.15.152:0/1285042493 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc90c014420 con 0x7fc9180f62a0 2023-12-06T04:00:37.384 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.384+0000 7fc91dbc2700 1 -- 172.21.15.152:0/1285042493 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc918107e80 con 0x7fc9180f62a0 2023-12-06T04:00:37.385 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.385+0000 7fc9157fa700 1 -- 172.21.15.152:0/1285042493 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc90c01c920 con 0x7fc9180f62a0 2023-12-06T04:00:37.385 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:37.386 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:37.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.386+0000 7fc91dbc2700 1 -- 172.21.15.152:0/1285042493 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc90802ff60 msgr2=0x7fc908032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:37.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.386+0000 7fc91dbc2700 1 --2- 172.21.15.152:0/1285042493 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc90802ff60 0x7fc908032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fc9040096a0 tx=0x7fc904006b40).stop 2023-12-06T04:00:37.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.386+0000 7fc91dbc2700 1 -- 172.21.15.152:0/1285042493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9180f62a0 msgr2=0x7fc9181064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:37.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.386+0000 7fc91dbc2700 1 --2- 172.21.15.152:0/1285042493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9180f62a0 0x7fc9181064b0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fc90c013510 tx=0x7fc90c00b040).stop 2023-12-06T04:00:37.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.386+0000 7fc91dbc2700 1 -- 172.21.15.152:0/1285042493 shutdown_connections 2023-12-06T04:00:37.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.386+0000 7fc91dbc2700 1 --2- 172.21.15.152:0/1285042493 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fc90802ff60 0x7fc908032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:37.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.386+0000 7fc91dbc2700 1 --2- 172.21.15.152:0/1285042493 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fc9180f62a0 0x7fc9181064b0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:37.390 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.386+0000 7fc91dbc2700 1 -- 172.21.15.152:0/1285042493 >> 172.21.15.152:0/1285042493 conn(0x7fc9180f1aa0 msgr2=0x7fc9180f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:37.391 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.386+0000 7fc91dbc2700 1 -- 172.21.15.152:0/1285042493 shutdown_connections 2023-12-06T04:00:37.391 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:37.386+0000 7fc91dbc2700 1 -- 172.21.15.152:0/1285042493 wait complete. 2023-12-06T04:00:37.391 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:37.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:36.458999+0000 mgr.smithi132.gokxja (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:37.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:37.385115+0000 mon.smithi132 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.152:0/1285042493' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:38.749 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:38.749 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:39.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:39 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:38.459311+0000 mgr.smithi132.gokxja (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:39.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.861+0000 7f6c15054700 1 -- 172.21.15.152:0/1892473817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c100f6510 msgr2=0x7f6c100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:39.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.861+0000 7f6c15054700 1 --2- 172.21.15.152:0/1892473817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c100f6510 0x7f6c100f6930 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f6bf8009fb0 tx=0x7f6bf8009b30).stop 2023-12-06T04:00:39.863 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.861+0000 7f6c15054700 1 -- 172.21.15.152:0/1892473817 shutdown_connections 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.861+0000 7f6c15054700 1 --2- 172.21.15.152:0/1892473817 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c100f6510 0x7f6c100f6930 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.861+0000 7f6c15054700 1 -- 172.21.15.152:0/1892473817 >> 172.21.15.152:0/1892473817 conn(0x7f6c100f1c70 msgr2=0x7f6c100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.861+0000 7f6c15054700 1 -- 172.21.15.152:0/1892473817 shutdown_connections 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.861+0000 7f6c15054700 1 -- 172.21.15.152:0/1892473817 wait complete. 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.862+0000 7f6c15054700 1 Processor -- start 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.862+0000 7f6c15054700 1 -- start start 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.862+0000 7f6c15054700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c100f6510 0x7f6c10106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.862+0000 7f6c15054700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c10106c20 con 0x7f6c100f6510 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.862+0000 7f6c0ed9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c100f6510 0x7f6c10106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.862+0000 7f6c0ed9d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c100f6510 0x7f6c10106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:46626/0 (socket says 172.21.15.152:46626) 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.862+0000 7f6c0ed9d700 1 -- 172.21.15.152:0/3616139414 learned_addr learned my addr 172.21.15.152:0/3616139414 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.862+0000 7f6c0ed9d700 1 -- 172.21.15.152:0/3616139414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf8005040 con 0x7f6c100f6510 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.863+0000 7f6c0ed9d700 1 --2- 172.21.15.152:0/3616139414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c100f6510 0x7f6c10106730 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f6bf8013510 tx=0x7f6bf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.863+0000 7f6c077fe700 1 -- 172.21.15.152:0/3616139414 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf8009320 con 0x7f6c100f6510 2023-12-06T04:00:39.864 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.863+0000 7f6c15054700 1 -- 172.21.15.152:0/3616139414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c10106e20 con 0x7f6c100f6510 2023-12-06T04:00:39.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.863+0000 7f6c077fe700 1 -- 172.21.15.152:0/3616139414 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6bf8013f00 con 0x7f6c100f6510 2023-12-06T04:00:39.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.863+0000 7f6c077fe700 1 -- 172.21.15.152:0/3616139414 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6bf8014d90 con 0x7f6c100f6510 2023-12-06T04:00:39.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.863+0000 7f6c15054700 1 -- 172.21.15.152:0/3616139414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c101072b0 con 0x7f6c100f6510 2023-12-06T04:00:39.865 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.864+0000 7f6c077fe700 1 -- 172.21.15.152:0/3616139414 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6bf8014420 con 0x7f6c100f6510 2023-12-06T04:00:39.866 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.864+0000 7f6c15054700 1 -- 172.21.15.152:0/3616139414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c10045ad0 con 0x7f6c100f6510 2023-12-06T04:00:39.866 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.864+0000 7f6c077fe700 1 --2- 172.21.15.152:0/3616139414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f6bfc02ffb0 0x7f6bfc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:39.866 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.864+0000 7f6c077fe700 1 -- 172.21.15.152:0/3616139414 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6bf8040fd0 con 0x7f6c100f6510 2023-12-06T04:00:39.866 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.864+0000 7f6c0e59c700 1 --2- 172.21.15.152:0/3616139414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f6bfc02ffb0 0x7f6bfc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:39.866 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.864+0000 7f6c0e59c700 1 --2- 172.21.15.152:0/3616139414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f6bfc02ffb0 0x7f6bfc032470 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f6c000096a0 tx=0x7f6c00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:39.869 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:39.868+0000 7f6c077fe700 1 -- 172.21.15.152:0/3616139414 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6bf806d2b0 con 0x7f6c100f6510 2023-12-06T04:00:40.077 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:40.076+0000 7f6c15054700 1 -- 172.21.15.152:0/3616139414 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c100fbf60 con 0x7f6c100f6510 2023-12-06T04:00:40.077 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:40.077+0000 7f6c077fe700 1 -- 172.21.15.152:0/3616139414 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6bf80095f0 con 0x7f6c100f6510 2023-12-06T04:00:40.078 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:40.078 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:40.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:40.078+0000 7f6c15054700 1 -- 172.21.15.152:0/3616139414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f6bfc02ffb0 msgr2=0x7f6bfc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:40.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:40.078+0000 7f6c15054700 1 --2- 172.21.15.152:0/3616139414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f6bfc02ffb0 0x7f6bfc032470 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f6c000096a0 tx=0x7f6c00006b40).stop 2023-12-06T04:00:40.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:40.078+0000 7f6c15054700 1 -- 172.21.15.152:0/3616139414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c100f6510 msgr2=0x7f6c10106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:40.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:40.078+0000 7f6c15054700 1 --2- 172.21.15.152:0/3616139414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c100f6510 0x7f6c10106730 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f6bf8013510 tx=0x7f6bf800b040).stop 2023-12-06T04:00:40.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:40.079+0000 7f6c15054700 1 -- 172.21.15.152:0/3616139414 shutdown_connections 2023-12-06T04:00:40.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:40.079+0000 7f6c15054700 1 --2- 172.21.15.152:0/3616139414 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f6bfc02ffb0 0x7f6bfc032470 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:40.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:40.079+0000 7f6c15054700 1 --2- 172.21.15.152:0/3616139414 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f6c100f6510 0x7f6c10106730 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:40.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:40.079+0000 7f6c15054700 1 -- 172.21.15.152:0/3616139414 >> 172.21.15.152:0/3616139414 conn(0x7f6c100f1c70 msgr2=0x7f6c100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:40.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:40.079+0000 7f6c15054700 1 -- 172.21.15.152:0/3616139414 shutdown_connections 2023-12-06T04:00:40.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:40.079+0000 7f6c15054700 1 -- 172.21.15.152:0/3616139414 wait complete. 2023-12-06T04:00:40.081 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:40.858 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:40 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:40.077509+0000 mon.smithi132 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.152:0/3616139414' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:41.454 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:41.454 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:41.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:41 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:40.459581+0000 mgr.smithi132.gokxja (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:42.591 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.590+0000 7f1de931d700 1 -- 172.21.15.152:0/2388620026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1de40f5410 msgr2=0x7f1de40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:42.591 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.590+0000 7f1de931d700 1 --2- 172.21.15.152:0/2388620026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1de40f5410 0x7f1de40f5830 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f1dcc009fb0 tx=0x7f1dcc009b30).stop 2023-12-06T04:00:42.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.590+0000 7f1de931d700 1 -- 172.21.15.152:0/2388620026 shutdown_connections 2023-12-06T04:00:42.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.590+0000 7f1de931d700 1 --2- 172.21.15.152:0/2388620026 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1de40f5410 0x7f1de40f5830 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:42.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.590+0000 7f1de931d700 1 -- 172.21.15.152:0/2388620026 >> 172.21.15.152:0/2388620026 conn(0x7f1de40f0b70 msgr2=0x7f1de40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:42.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.590+0000 7f1de931d700 1 -- 172.21.15.152:0/2388620026 shutdown_connections 2023-12-06T04:00:42.592 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.590+0000 7f1de931d700 1 -- 172.21.15.152:0/2388620026 wait complete. 2023-12-06T04:00:42.594 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.591+0000 7f1de931d700 1 Processor -- start 2023-12-06T04:00:42.594 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.591+0000 7f1de931d700 1 -- start start 2023-12-06T04:00:42.595 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.591+0000 7f1de931d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1de40f5410 0x7f1de4106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:42.595 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.591+0000 7f1de931d700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1de4106c10 con 0x7f1de40f5410 2023-12-06T04:00:42.595 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.592+0000 7f1de2ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1de40f5410 0x7f1de4106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:42.595 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.592+0000 7f1de2ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1de40f5410 0x7f1de4106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:46642/0 (socket says 172.21.15.152:46642) 2023-12-06T04:00:42.595 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.592+0000 7f1de2ffd700 1 -- 172.21.15.152:0/3894690769 learned_addr learned my addr 172.21.15.152:0/3894690769 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:42.595 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.592+0000 7f1de2ffd700 1 -- 172.21.15.152:0/3894690769 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dcc005040 con 0x7f1de40f5410 2023-12-06T04:00:42.595 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.592+0000 7f1de2ffd700 1 --2- 172.21.15.152:0/3894690769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1de40f5410 0x7f1de4106720 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f1dcc013510 tx=0x7f1dcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:42.595 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.592+0000 7f1ddb7fe700 1 -- 172.21.15.152:0/3894690769 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1dcc009320 con 0x7f1de40f5410 2023-12-06T04:00:42.595 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.592+0000 7f1de931d700 1 -- 172.21.15.152:0/3894690769 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1de4106e10 con 0x7f1de40f5410 2023-12-06T04:00:42.595 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.592+0000 7f1ddb7fe700 1 -- 172.21.15.152:0/3894690769 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1dcc013f00 con 0x7f1de40f5410 2023-12-06T04:00:42.595 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.592+0000 7f1ddb7fe700 1 -- 172.21.15.152:0/3894690769 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1dcc014d90 con 0x7f1de40f5410 2023-12-06T04:00:42.595 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.592+0000 7f1de931d700 1 -- 172.21.15.152:0/3894690769 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1de41072a0 con 0x7f1de40f5410 2023-12-06T04:00:42.595 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.593+0000 7f1ddb7fe700 1 -- 172.21.15.152:0/3894690769 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1dcc0095f0 con 0x7f1de40f5410 2023-12-06T04:00:42.596 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.593+0000 7f1de931d700 1 -- 172.21.15.152:0/3894690769 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1de4045ad0 con 0x7f1de40f5410 2023-12-06T04:00:42.596 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.593+0000 7f1ddb7fe700 1 --2- 172.21.15.152:0/3894690769 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1dd002ff60 0x7f1dd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:42.596 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.593+0000 7f1ddb7fe700 1 -- 172.21.15.152:0/3894690769 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1dcc041000 con 0x7f1de40f5410 2023-12-06T04:00:42.596 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.593+0000 7f1de27fc700 1 --2- 172.21.15.152:0/3894690769 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1dd002ff60 0x7f1dd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:42.596 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.594+0000 7f1de27fc700 1 --2- 172.21.15.152:0/3894690769 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1dd002ff60 0x7f1dd0032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f1dd40096a0 tx=0x7f1dd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:42.599 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.598+0000 7f1ddb7fe700 1 -- 172.21.15.152:0/3894690769 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1dcc014420 con 0x7f1de40f5410 2023-12-06T04:00:42.809 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.808+0000 7f1de931d700 1 -- 172.21.15.152:0/3894690769 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1de4108150 con 0x7f1de40f5410 2023-12-06T04:00:42.810 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.809+0000 7f1ddb7fe700 1 -- 172.21.15.152:0/3894690769 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1dcc010230 con 0x7f1de40f5410 2023-12-06T04:00:42.811 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:42.811 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:42.812 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.811+0000 7f1de931d700 1 -- 172.21.15.152:0/3894690769 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1dd002ff60 msgr2=0x7f1dd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:42.812 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.811+0000 7f1de931d700 1 --2- 172.21.15.152:0/3894690769 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1dd002ff60 0x7f1dd0032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f1dd40096a0 tx=0x7f1dd4006b40).stop 2023-12-06T04:00:42.812 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.811+0000 7f1de931d700 1 -- 172.21.15.152:0/3894690769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1de40f5410 msgr2=0x7f1de4106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:42.813 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.811+0000 7f1de931d700 1 --2- 172.21.15.152:0/3894690769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1de40f5410 0x7f1de4106720 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f1dcc013510 tx=0x7f1dcc00b040).stop 2023-12-06T04:00:42.813 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.811+0000 7f1de931d700 1 -- 172.21.15.152:0/3894690769 shutdown_connections 2023-12-06T04:00:42.813 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.811+0000 7f1de931d700 1 --2- 172.21.15.152:0/3894690769 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1dd002ff60 0x7f1dd0032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:42.813 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.811+0000 7f1de931d700 1 --2- 172.21.15.152:0/3894690769 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1de40f5410 0x7f1de4106720 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:42.813 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.811+0000 7f1de931d700 1 -- 172.21.15.152:0/3894690769 >> 172.21.15.152:0/3894690769 conn(0x7f1de40f0b70 msgr2=0x7f1de405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:42.813 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.811+0000 7f1de931d700 1 -- 172.21.15.152:0/3894690769 shutdown_connections 2023-12-06T04:00:42.814 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:42.811+0000 7f1de931d700 1 -- 172.21.15.152:0/3894690769 wait complete. 2023-12-06T04:00:42.814 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:43.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:42.459917+0000 mgr.smithi132.gokxja (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:43.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:42.809735+0000 mon.smithi132 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.152:0/3894690769' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:44.201 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:44.202 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:45.280 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.279+0000 7fb2b339e700 1 -- 172.21.15.152:0/1509322748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2ac060180 msgr2=0x7fb2ac0f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:45.281 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.279+0000 7fb2b339e700 1 --2- 172.21.15.152:0/1509322748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2ac060180 0x7fb2ac0f8f80 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fb2a8009fb0 tx=0x7fb2a8009b30).stop 2023-12-06T04:00:45.281 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.279+0000 7fb2b339e700 1 -- 172.21.15.152:0/1509322748 shutdown_connections 2023-12-06T04:00:45.281 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.279+0000 7fb2b339e700 1 --2- 172.21.15.152:0/1509322748 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2ac060180 0x7fb2ac0f8f80 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:45.281 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.279+0000 7fb2b339e700 1 -- 172.21.15.152:0/1509322748 >> 172.21.15.152:0/1509322748 conn(0x7fb2ac0f09a0 msgr2=0x7fb2ac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:45.281 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.279+0000 7fb2b339e700 1 -- 172.21.15.152:0/1509322748 shutdown_connections 2023-12-06T04:00:45.281 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.279+0000 7fb2b339e700 1 -- 172.21.15.152:0/1509322748 wait complete. 2023-12-06T04:00:45.281 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.279+0000 7fb2b339e700 1 Processor -- start 2023-12-06T04:00:45.281 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.279+0000 7fb2b339e700 1 -- start start 2023-12-06T04:00:45.281 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.279+0000 7fb2b339e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2ac060180 0x7fb2ac0f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:45.281 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.279+0000 7fb2b339e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2ac0f4e80 con 0x7fb2ac060180 2023-12-06T04:00:45.281 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.280+0000 7fb2b239c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2ac060180 0x7fb2ac0f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:45.281 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.280+0000 7fb2b239c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2ac060180 0x7fb2ac0f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:46660/0 (socket says 172.21.15.152:46660) 2023-12-06T04:00:45.282 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.280+0000 7fb2b239c700 1 -- 172.21.15.152:0/4219275176 learned_addr learned my addr 172.21.15.152:0/4219275176 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:45.282 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.280+0000 7fb2b239c700 1 -- 172.21.15.152:0/4219275176 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a8005040 con 0x7fb2ac060180 2023-12-06T04:00:45.282 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.280+0000 7fb2b239c700 1 --2- 172.21.15.152:0/4219275176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2ac060180 0x7fb2ac0f6a00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fb2a8014000 tx=0x7fb2a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:45.282 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.280+0000 7fb2a2ffd700 1 -- 172.21.15.152:0/4219275176 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2a80095f0 con 0x7fb2ac060180 2023-12-06T04:00:45.282 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.280+0000 7fb2b339e700 1 -- 172.21.15.152:0/4219275176 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ac0f5020 con 0x7fb2ac060180 2023-12-06T04:00:45.282 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.280+0000 7fb2a2ffd700 1 -- 172.21.15.152:0/4219275176 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2a8007e70 con 0x7fb2ac060180 2023-12-06T04:00:45.282 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.280+0000 7fb2a2ffd700 1 -- 172.21.15.152:0/4219275176 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2a8014ea0 con 0x7fb2ac060180 2023-12-06T04:00:45.282 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.280+0000 7fb2b339e700 1 -- 172.21.15.152:0/4219275176 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2ac0f54b0 con 0x7fb2ac060180 2023-12-06T04:00:45.283 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.281+0000 7fb2b339e700 1 -- 172.21.15.152:0/4219275176 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2ac059100 con 0x7fb2ac060180 2023-12-06T04:00:45.283 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.281+0000 7fb2a2ffd700 1 -- 172.21.15.152:0/4219275176 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2a8025070 con 0x7fb2ac060180 2023-12-06T04:00:45.283 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.281+0000 7fb2a2ffd700 1 --2- 172.21.15.152:0/4219275176 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb29802ffb0 0x7fb298032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:45.283 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.281+0000 7fb2a2ffd700 1 -- 172.21.15.152:0/4219275176 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb2a80408d0 con 0x7fb2ac060180 2023-12-06T04:00:45.283 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.281+0000 7fb2b1b9b700 1 --2- 172.21.15.152:0/4219275176 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb29802ffb0 0x7fb298032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:45.283 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.282+0000 7fb2b1b9b700 1 --2- 172.21.15.152:0/4219275176 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb29802ffb0 0x7fb298032470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fb29c0096a0 tx=0x7fb29c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:45.286 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.286+0000 7fb2a2ffd700 1 -- 172.21.15.152:0/4219275176 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2a80079c0 con 0x7fb2ac060180 2023-12-06T04:00:45.493 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.492+0000 7fb2b339e700 1 -- 172.21.15.152:0/4219275176 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2ac045ad0 con 0x7fb2ac060180 2023-12-06T04:00:45.494 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.493+0000 7fb2a2ffd700 1 -- 172.21.15.152:0/4219275176 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb2a800f300 con 0x7fb2ac060180 2023-12-06T04:00:45.494 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:45.494 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:45.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.495+0000 7fb2b339e700 1 -- 172.21.15.152:0/4219275176 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb29802ffb0 msgr2=0x7fb298032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:45.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.495+0000 7fb2b339e700 1 --2- 172.21.15.152:0/4219275176 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb29802ffb0 0x7fb298032470 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fb29c0096a0 tx=0x7fb29c006b40).stop 2023-12-06T04:00:45.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.495+0000 7fb2b339e700 1 -- 172.21.15.152:0/4219275176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2ac060180 msgr2=0x7fb2ac0f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:45.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.495+0000 7fb2b339e700 1 --2- 172.21.15.152:0/4219275176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2ac060180 0x7fb2ac0f6a00 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fb2a8014000 tx=0x7fb2a800b040).stop 2023-12-06T04:00:45.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.495+0000 7fb2b339e700 1 -- 172.21.15.152:0/4219275176 shutdown_connections 2023-12-06T04:00:45.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.495+0000 7fb2b339e700 1 --2- 172.21.15.152:0/4219275176 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb29802ffb0 0x7fb298032470 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:45.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.495+0000 7fb2b339e700 1 --2- 172.21.15.152:0/4219275176 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb2ac060180 0x7fb2ac0f6a00 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:45.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.495+0000 7fb2b339e700 1 -- 172.21.15.152:0/4219275176 >> 172.21.15.152:0/4219275176 conn(0x7fb2ac0f09a0 msgr2=0x7fb2ac0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:45.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.495+0000 7fb2b339e700 1 -- 172.21.15.152:0/4219275176 shutdown_connections 2023-12-06T04:00:45.496 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:45.495+0000 7fb2b339e700 1 -- 172.21.15.152:0/4219275176 wait complete. 2023-12-06T04:00:45.497 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:45.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:45 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:44.460323+0000 mgr.smithi132.gokxja (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:45.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:45 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:45.493693+0000 mon.smithi132 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.152:0/4219275176' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:46.822 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:46.822 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:47.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:47 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:46.460677+0000 mgr.smithi132.gokxja (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:47.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.932+0000 7ff3a764e700 1 -- 172.21.15.152:0/1340125340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a00f4740 msgr2=0x7ff3a00f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:47.934 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.932+0000 7ff3a764e700 1 --2- 172.21.15.152:0/1340125340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a00f4740 0x7ff3a00f4b60 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7ff390009fb0 tx=0x7ff390009b30).stop 2023-12-06T04:00:47.935 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.932+0000 7ff3a764e700 1 -- 172.21.15.152:0/1340125340 shutdown_connections 2023-12-06T04:00:47.935 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.932+0000 7ff3a764e700 1 --2- 172.21.15.152:0/1340125340 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a00f4740 0x7ff3a00f4b60 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:47.935 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.932+0000 7ff3a764e700 1 -- 172.21.15.152:0/1340125340 >> 172.21.15.152:0/1340125340 conn(0x7ff3a00f0260 msgr2=0x7ff3a00f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:47.935 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.932+0000 7ff3a764e700 1 -- 172.21.15.152:0/1340125340 shutdown_connections 2023-12-06T04:00:47.935 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.932+0000 7ff3a764e700 1 -- 172.21.15.152:0/1340125340 wait complete. 2023-12-06T04:00:47.935 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.933+0000 7ff3a764e700 1 Processor -- start 2023-12-06T04:00:47.935 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.933+0000 7ff3a764e700 1 -- start start 2023-12-06T04:00:47.935 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.933+0000 7ff3a764e700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a00f4740 0x7ff3a01066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:47.935 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.933+0000 7ff3a764e700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3a0106bc0 con 0x7ff3a00f4740 2023-12-06T04:00:47.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.933+0000 7ff3a53ea700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a00f4740 0x7ff3a01066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:47.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.933+0000 7ff3a53ea700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a00f4740 0x7ff3a01066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:46680/0 (socket says 172.21.15.152:46680) 2023-12-06T04:00:47.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.933+0000 7ff3a53ea700 1 -- 172.21.15.152:0/2476631636 learned_addr learned my addr 172.21.15.152:0/2476631636 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:47.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.933+0000 7ff3a53ea700 1 -- 172.21.15.152:0/2476631636 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff390005040 con 0x7ff3a00f4740 2023-12-06T04:00:47.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.934+0000 7ff3a53ea700 1 --2- 172.21.15.152:0/2476631636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a00f4740 0x7ff3a01066d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7ff390013510 tx=0x7ff39000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:47.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.934+0000 7ff395ffb700 1 -- 172.21.15.152:0/2476631636 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff390009320 con 0x7ff3a00f4740 2023-12-06T04:00:47.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.934+0000 7ff3a764e700 1 -- 172.21.15.152:0/2476631636 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3a0106dc0 con 0x7ff3a00f4740 2023-12-06T04:00:47.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.934+0000 7ff395ffb700 1 -- 172.21.15.152:0/2476631636 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff390013f00 con 0x7ff3a00f4740 2023-12-06T04:00:47.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.934+0000 7ff395ffb700 1 -- 172.21.15.152:0/2476631636 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff390014d90 con 0x7ff3a00f4740 2023-12-06T04:00:47.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.934+0000 7ff3a764e700 1 -- 172.21.15.152:0/2476631636 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3a0107250 con 0x7ff3a00f4740 2023-12-06T04:00:47.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.935+0000 7ff395ffb700 1 -- 172.21.15.152:0/2476631636 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3900263a0 con 0x7ff3a00f4740 2023-12-06T04:00:47.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.935+0000 7ff395ffb700 1 --2- 172.21.15.152:0/2476631636 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff38c02ff60 0x7ff38c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:47.937 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.935+0000 7ff395ffb700 1 -- 172.21.15.152:0/2476631636 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff390040fc0 con 0x7ff3a00f4740 2023-12-06T04:00:47.938 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.935+0000 7ff3a764e700 1 -- 172.21.15.152:0/2476631636 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3a0045ad0 con 0x7ff3a00f4740 2023-12-06T04:00:47.938 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.935+0000 7ff3a4be9700 1 --2- 172.21.15.152:0/2476631636 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff38c02ff60 0x7ff38c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:47.938 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.936+0000 7ff3a4be9700 1 --2- 172.21.15.152:0/2476631636 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff38c02ff60 0x7ff38c032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff39c0096a0 tx=0x7ff39c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:47.941 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:47.940+0000 7ff395ffb700 1 -- 172.21.15.152:0/2476631636 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff390012920 con 0x7ff3a00f4740 2023-12-06T04:00:48.146 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:48.145+0000 7ff3a764e700 1 -- 172.21.15.152:0/2476631636 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3a0107f90 con 0x7ff3a00f4740 2023-12-06T04:00:48.147 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:48.145+0000 7ff395ffb700 1 -- 172.21.15.152:0/2476631636 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff39000f340 con 0x7ff3a00f4740 2023-12-06T04:00:48.147 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:48.147 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:48.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:48.147+0000 7ff3a764e700 1 -- 172.21.15.152:0/2476631636 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff38c02ff60 msgr2=0x7ff38c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:48.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:48.147+0000 7ff3a764e700 1 --2- 172.21.15.152:0/2476631636 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff38c02ff60 0x7ff38c032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7ff39c0096a0 tx=0x7ff39c006b40).stop 2023-12-06T04:00:48.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:48.147+0000 7ff3a764e700 1 -- 172.21.15.152:0/2476631636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a00f4740 msgr2=0x7ff3a01066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:48.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:48.147+0000 7ff3a764e700 1 --2- 172.21.15.152:0/2476631636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a00f4740 0x7ff3a01066d0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7ff390013510 tx=0x7ff39000b040).stop 2023-12-06T04:00:48.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:48.147+0000 7ff3a764e700 1 -- 172.21.15.152:0/2476631636 shutdown_connections 2023-12-06T04:00:48.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:48.147+0000 7ff3a764e700 1 --2- 172.21.15.152:0/2476631636 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ff38c02ff60 0x7ff38c032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:48.149 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:48.147+0000 7ff3a764e700 1 --2- 172.21.15.152:0/2476631636 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ff3a00f4740 0x7ff3a01066d0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:48.150 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:48.147+0000 7ff3a764e700 1 -- 172.21.15.152:0/2476631636 >> 172.21.15.152:0/2476631636 conn(0x7ff3a00f0260 msgr2=0x7ff3a00f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:48.150 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:48.147+0000 7ff3a764e700 1 -- 172.21.15.152:0/2476631636 shutdown_connections 2023-12-06T04:00:48.150 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:48.147+0000 7ff3a764e700 1 -- 172.21.15.152:0/2476631636 wait complete. 2023-12-06T04:00:48.150 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:48.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:48 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:48.146705+0000 mon.smithi132 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.152:0/2476631636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:49.548 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:49.549 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:49.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:49 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:48.460989+0000 mgr.smithi132.gokxja (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:50.643 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.641+0000 7f29d95cf700 1 -- 172.21.15.152:0/1250002360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29d4060280 msgr2=0x7f29d40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:50.643 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.641+0000 7f29d95cf700 1 --2- 172.21.15.152:0/1250002360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29d4060280 0x7f29d40f90f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f29c8009fb0 tx=0x7f29c8009b30).stop 2023-12-06T04:00:50.643 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.641+0000 7f29d95cf700 1 -- 172.21.15.152:0/1250002360 shutdown_connections 2023-12-06T04:00:50.643 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.641+0000 7f29d95cf700 1 --2- 172.21.15.152:0/1250002360 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29d4060280 0x7f29d40f90f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:50.643 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.641+0000 7f29d95cf700 1 -- 172.21.15.152:0/1250002360 >> 172.21.15.152:0/1250002360 conn(0x7f29d40f0b70 msgr2=0x7f29d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:50.643 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.641+0000 7f29d95cf700 1 -- 172.21.15.152:0/1250002360 shutdown_connections 2023-12-06T04:00:50.643 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.641+0000 7f29d95cf700 1 -- 172.21.15.152:0/1250002360 wait complete. 2023-12-06T04:00:50.645 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.642+0000 7f29d95cf700 1 Processor -- start 2023-12-06T04:00:50.645 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.642+0000 7f29d95cf700 1 -- start start 2023-12-06T04:00:50.645 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.642+0000 7f29d95cf700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29d4060280 0x7f29d41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:50.645 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.642+0000 7f29d95cf700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29d4106bd0 con 0x7f29d4060280 2023-12-06T04:00:50.645 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.642+0000 7f29d2ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29d4060280 0x7f29d41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:50.645 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.642+0000 7f29d2ffd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29d4060280 0x7f29d41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:34536/0 (socket says 172.21.15.152:34536) 2023-12-06T04:00:50.645 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.642+0000 7f29d2ffd700 1 -- 172.21.15.152:0/3518162508 learned_addr learned my addr 172.21.15.152:0/3518162508 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:50.645 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.643+0000 7f29d2ffd700 1 -- 172.21.15.152:0/3518162508 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c8005040 con 0x7f29d4060280 2023-12-06T04:00:50.645 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.643+0000 7f29d2ffd700 1 --2- 172.21.15.152:0/3518162508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29d4060280 0x7f29d41066e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f29c8013510 tx=0x7f29c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:50.645 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.643+0000 7f29bb7fe700 1 -- 172.21.15.152:0/3518162508 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29c8009320 con 0x7f29d4060280 2023-12-06T04:00:50.646 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.643+0000 7f29d95cf700 1 -- 172.21.15.152:0/3518162508 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29d4106dd0 con 0x7f29d4060280 2023-12-06T04:00:50.646 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.643+0000 7f29bb7fe700 1 -- 172.21.15.152:0/3518162508 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29c8013f00 con 0x7f29d4060280 2023-12-06T04:00:50.646 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.643+0000 7f29bb7fe700 1 -- 172.21.15.152:0/3518162508 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29c8014dc0 con 0x7f29d4060280 2023-12-06T04:00:50.646 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.643+0000 7f29d95cf700 1 -- 172.21.15.152:0/3518162508 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29d4107260 con 0x7f29d4060280 2023-12-06T04:00:50.647 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.644+0000 7f29bb7fe700 1 -- 172.21.15.152:0/3518162508 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29c80095f0 con 0x7f29d4060280 2023-12-06T04:00:50.647 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.644+0000 7f29d95cf700 1 -- 172.21.15.152:0/3518162508 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29d4045ad0 con 0x7f29d4060280 2023-12-06T04:00:50.647 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.644+0000 7f29bb7fe700 1 --2- 172.21.15.152:0/3518162508 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f29bc02ff60 0x7f29bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:50.647 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.644+0000 7f29bb7fe700 1 -- 172.21.15.152:0/3518162508 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29c8041000 con 0x7f29d4060280 2023-12-06T04:00:50.647 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.644+0000 7f29d27fc700 1 --2- 172.21.15.152:0/3518162508 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f29bc02ff60 0x7f29bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:50.647 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.645+0000 7f29d27fc700 1 --2- 172.21.15.152:0/3518162508 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f29bc02ff60 0x7f29bc032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f29c40096a0 tx=0x7f29c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:50.650 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.649+0000 7f29bb7fe700 1 -- 172.21.15.152:0/3518162508 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29c801c790 con 0x7f29d4060280 2023-12-06T04:00:50.857 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.855+0000 7f29d95cf700 1 -- 172.21.15.152:0/3518162508 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29d4108110 con 0x7f29d4060280 2023-12-06T04:00:50.857 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.856+0000 7f29bb7fe700 1 -- 172.21.15.152:0/3518162508 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f29c8010230 con 0x7f29d4060280 2023-12-06T04:00:50.858 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:50.858 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:50.859 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.857+0000 7f29d95cf700 1 -- 172.21.15.152:0/3518162508 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f29bc02ff60 msgr2=0x7f29bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:50.859 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.857+0000 7f29d95cf700 1 --2- 172.21.15.152:0/3518162508 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f29bc02ff60 0x7f29bc032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f29c40096a0 tx=0x7f29c4006b40).stop 2023-12-06T04:00:50.859 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.857+0000 7f29d95cf700 1 -- 172.21.15.152:0/3518162508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29d4060280 msgr2=0x7f29d41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:50.859 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.857+0000 7f29d95cf700 1 --2- 172.21.15.152:0/3518162508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29d4060280 0x7f29d41066e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f29c8013510 tx=0x7f29c800b040).stop 2023-12-06T04:00:50.859 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.857+0000 7f29d95cf700 1 -- 172.21.15.152:0/3518162508 shutdown_connections 2023-12-06T04:00:50.860 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.857+0000 7f29d95cf700 1 --2- 172.21.15.152:0/3518162508 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f29bc02ff60 0x7f29bc032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:50.860 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.857+0000 7f29d95cf700 1 --2- 172.21.15.152:0/3518162508 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f29d4060280 0x7f29d41066e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:50.860 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.857+0000 7f29d95cf700 1 -- 172.21.15.152:0/3518162508 >> 172.21.15.152:0/3518162508 conn(0x7f29d40f0b70 msgr2=0x7f29d40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:50.860 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.857+0000 7f29d95cf700 1 -- 172.21.15.152:0/3518162508 shutdown_connections 2023-12-06T04:00:50.860 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:50.858+0000 7f29d95cf700 1 -- 172.21.15.152:0/3518162508 wait complete. 2023-12-06T04:00:50.860 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:51.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:51 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:50.461363+0000 mgr.smithi132.gokxja (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:51.857 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:51 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:50.857399+0000 mon.smithi132 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.152:0/3518162508' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:52.180 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:52.181 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:53.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.334+0000 7fa8f8eae700 1 -- 172.21.15.152:0/1219434818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8f40f62a0 msgr2=0x7fa8f40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:53.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.334+0000 7fa8f8eae700 1 --2- 172.21.15.152:0/1219434818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8f40f62a0 0x7fa8f40f66c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7fa8e4009fb0 tx=0x7fa8e4009b30).stop 2023-12-06T04:00:53.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.334+0000 7fa8f8eae700 1 -- 172.21.15.152:0/1219434818 shutdown_connections 2023-12-06T04:00:53.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.334+0000 7fa8f8eae700 1 --2- 172.21.15.152:0/1219434818 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8f40f62a0 0x7fa8f40f66c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:53.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.334+0000 7fa8f8eae700 1 -- 172.21.15.152:0/1219434818 >> 172.21.15.152:0/1219434818 conn(0x7fa8f40f1aa0 msgr2=0x7fa8f40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:53.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.334+0000 7fa8f8eae700 1 -- 172.21.15.152:0/1219434818 shutdown_connections 2023-12-06T04:00:53.337 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.334+0000 7fa8f8eae700 1 -- 172.21.15.152:0/1219434818 wait complete. 2023-12-06T04:00:53.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.334+0000 7fa8f8eae700 1 Processor -- start 2023-12-06T04:00:53.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.334+0000 7fa8f8eae700 1 -- start start 2023-12-06T04:00:53.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.334+0000 7fa8f8eae700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8f40f62a0 0x7fa8f41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:53.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.335+0000 7fa8f8eae700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8f41069a0 con 0x7fa8f40f62a0 2023-12-06T04:00:53.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.335+0000 7fa8f37fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8f40f62a0 0x7fa8f41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:53.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.335+0000 7fa8f37fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8f40f62a0 0x7fa8f41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:34556/0 (socket says 172.21.15.152:34556) 2023-12-06T04:00:53.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.335+0000 7fa8f37fe700 1 -- 172.21.15.152:0/2232354487 learned_addr learned my addr 172.21.15.152:0/2232354487 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:53.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.335+0000 7fa8f37fe700 1 -- 172.21.15.152:0/2232354487 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8e4005040 con 0x7fa8f40f62a0 2023-12-06T04:00:53.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.335+0000 7fa8f37fe700 1 --2- 172.21.15.152:0/2232354487 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8f40f62a0 0x7fa8f41064b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fa8e4009830 tx=0x7fa8e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:53.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.335+0000 7fa8dbfff700 1 -- 172.21.15.152:0/2232354487 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8e4009320 con 0x7fa8f40f62a0 2023-12-06T04:00:53.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.335+0000 7fa8f8eae700 1 -- 172.21.15.152:0/2232354487 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8f4106ba0 con 0x7fa8f40f62a0 2023-12-06T04:00:53.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.335+0000 7fa8dbfff700 1 -- 172.21.15.152:0/2232354487 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8e40136c0 con 0x7fa8f40f62a0 2023-12-06T04:00:53.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.335+0000 7fa8dbfff700 1 -- 172.21.15.152:0/2232354487 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa8e401ee90 con 0x7fa8f40f62a0 2023-12-06T04:00:53.338 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.335+0000 7fa8f8eae700 1 -- 172.21.15.152:0/2232354487 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8f4107030 con 0x7fa8f40f62a0 2023-12-06T04:00:53.339 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.336+0000 7fa8dbfff700 1 -- 172.21.15.152:0/2232354487 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa8e401e420 con 0x7fa8f40f62a0 2023-12-06T04:00:53.339 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.336+0000 7fa8dbfff700 1 --2- 172.21.15.152:0/2232354487 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa8dc02ff10 0x7fa8dc0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:53.339 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.336+0000 7fa8f8eae700 1 -- 172.21.15.152:0/2232354487 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8f4045ad0 con 0x7fa8f40f62a0 2023-12-06T04:00:53.339 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.336+0000 7fa8dbfff700 1 -- 172.21.15.152:0/2232354487 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa8e4041050 con 0x7fa8f40f62a0 2023-12-06T04:00:53.339 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.336+0000 7fa8f2ffd700 1 --2- 172.21.15.152:0/2232354487 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa8dc02ff10 0x7fa8dc0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:53.340 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.337+0000 7fa8f2ffd700 1 --2- 172.21.15.152:0/2232354487 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa8dc02ff10 0x7fa8dc0323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fa8e80096a0 tx=0x7fa8e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:53.342 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.341+0000 7fa8dbfff700 1 -- 172.21.15.152:0/2232354487 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8e401c790 con 0x7fa8f40f62a0 2023-12-06T04:00:53.549 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.547+0000 7fa8f8eae700 1 -- 172.21.15.152:0/2232354487 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa8f4107e80 con 0x7fa8f40f62a0 2023-12-06T04:00:53.550 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.548+0000 7fa8dbfff700 1 -- 172.21.15.152:0/2232354487 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa8e400e2a0 con 0x7fa8f40f62a0 2023-12-06T04:00:53.550 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:53.550 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:53.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.550+0000 7fa8f8eae700 1 -- 172.21.15.152:0/2232354487 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa8dc02ff10 msgr2=0x7fa8dc0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:53.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.550+0000 7fa8f8eae700 1 --2- 172.21.15.152:0/2232354487 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa8dc02ff10 0x7fa8dc0323d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fa8e80096a0 tx=0x7fa8e8006b40).stop 2023-12-06T04:00:53.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.550+0000 7fa8f8eae700 1 -- 172.21.15.152:0/2232354487 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8f40f62a0 msgr2=0x7fa8f41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:53.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.550+0000 7fa8f8eae700 1 --2- 172.21.15.152:0/2232354487 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8f40f62a0 0x7fa8f41064b0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fa8e4009830 tx=0x7fa8e400b040).stop 2023-12-06T04:00:53.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.550+0000 7fa8f8eae700 1 -- 172.21.15.152:0/2232354487 shutdown_connections 2023-12-06T04:00:53.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.550+0000 7fa8f8eae700 1 --2- 172.21.15.152:0/2232354487 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa8dc02ff10 0x7fa8dc0323d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:53.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.550+0000 7fa8f8eae700 1 --2- 172.21.15.152:0/2232354487 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa8f40f62a0 0x7fa8f41064b0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:53.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.550+0000 7fa8f8eae700 1 -- 172.21.15.152:0/2232354487 >> 172.21.15.152:0/2232354487 conn(0x7fa8f40f1aa0 msgr2=0x7fa8f40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:53.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.550+0000 7fa8f8eae700 1 -- 172.21.15.152:0/2232354487 shutdown_connections 2023-12-06T04:00:53.552 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:53.550+0000 7fa8f8eae700 1 -- 172.21.15.152:0/2232354487 wait complete. 2023-12-06T04:00:53.553 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:53.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:53 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:52.461743+0000 mgr.smithi132.gokxja (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:53.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:53 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:53.549716+0000 mon.smithi132 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.152:0/2232354487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:54.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:54.524138+0000 mon.smithi132 (mon.0) 306 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/mirror_snapshot_schedule"}]: dispatch 2023-12-06T04:00:54.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:54 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:54.531910+0000 mon.smithi132 (mon.0) 307 : audit [INF] from='mgr.14162 172.21.15.132:0/3659673009' entity='mgr.smithi132.gokxja' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi132.gokxja/trash_purge_schedule"}]: dispatch 2023-12-06T04:00:54.935 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:54.935 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:55.856 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:55 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:54.462075+0000 mgr.smithi132.gokxja (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:56.077 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.075+0000 7f91b5b06700 1 -- 172.21.15.152:0/4002656621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91b0060200 msgr2=0x7f91b00fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:56.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.075+0000 7f91b5b06700 1 --2- 172.21.15.152:0/4002656621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91b0060200 0x7f91b00fb570 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f91a0009fb0 tx=0x7f91a0009b30).stop 2023-12-06T04:00:56.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.075+0000 7f91b5b06700 1 -- 172.21.15.152:0/4002656621 shutdown_connections 2023-12-06T04:00:56.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.075+0000 7f91b5b06700 1 --2- 172.21.15.152:0/4002656621 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91b0060200 0x7f91b00fb570 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:56.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.075+0000 7f91b5b06700 1 -- 172.21.15.152:0/4002656621 >> 172.21.15.152:0/4002656621 conn(0x7f91b00f0b70 msgr2=0x7f91b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:56.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.075+0000 7f91b5b06700 1 -- 172.21.15.152:0/4002656621 shutdown_connections 2023-12-06T04:00:56.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.075+0000 7f91b5b06700 1 -- 172.21.15.152:0/4002656621 wait complete. 2023-12-06T04:00:56.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.076+0000 7f91b5b06700 1 Processor -- start 2023-12-06T04:00:56.078 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.076+0000 7f91b5b06700 1 -- start start 2023-12-06T04:00:56.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.076+0000 7f91b5b06700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91b0060200 0x7f91b01088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:56.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.076+0000 7f91b5b06700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91b0108dc0 con 0x7f91b0060200 2023-12-06T04:00:56.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.076+0000 7f91af7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91b0060200 0x7f91b01088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:56.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.076+0000 7f91af7fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91b0060200 0x7f91b01088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:34572/0 (socket says 172.21.15.152:34572) 2023-12-06T04:00:56.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.077+0000 7f91af7fe700 1 -- 172.21.15.152:0/886284045 learned_addr learned my addr 172.21.15.152:0/886284045 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:56.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.077+0000 7f91af7fe700 1 -- 172.21.15.152:0/886284045 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a0005040 con 0x7f91b0060200 2023-12-06T04:00:56.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.077+0000 7f91af7fe700 1 --2- 172.21.15.152:0/886284045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91b0060200 0x7f91b01088d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f91a0013510 tx=0x7f91a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:56.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.077+0000 7f9197fff700 1 -- 172.21.15.152:0/886284045 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91a0009320 con 0x7f91b0060200 2023-12-06T04:00:56.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.077+0000 7f9197fff700 1 -- 172.21.15.152:0/886284045 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f91a0013f00 con 0x7f91b0060200 2023-12-06T04:00:56.080 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.077+0000 7f91b5b06700 1 -- 172.21.15.152:0/886284045 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91b0108fc0 con 0x7f91b0060200 2023-12-06T04:00:56.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.077+0000 7f9197fff700 1 -- 172.21.15.152:0/886284045 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91a0014dc0 con 0x7f91b0060200 2023-12-06T04:00:56.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.077+0000 7f91b5b06700 1 -- 172.21.15.152:0/886284045 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91b0109450 con 0x7f91b0060200 2023-12-06T04:00:56.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.078+0000 7f9197fff700 1 -- 172.21.15.152:0/886284045 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91a001d3d0 con 0x7f91b0060200 2023-12-06T04:00:56.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.078+0000 7f91b5b06700 1 -- 172.21.15.152:0/886284045 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91b0045ad0 con 0x7f91b0060200 2023-12-06T04:00:56.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.078+0000 7f9197fff700 1 --2- 172.21.15.152:0/886284045 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f919802ff60 0x7f9198032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:56.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.078+0000 7f9197fff700 1 -- 172.21.15.152:0/886284045 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f91a00410f0 con 0x7f91b0060200 2023-12-06T04:00:56.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.078+0000 7f91aeffd700 1 --2- 172.21.15.152:0/886284045 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f919802ff60 0x7f9198032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:56.081 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.079+0000 7f91aeffd700 1 --2- 172.21.15.152:0/886284045 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f919802ff60 0x7f9198032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f91a40096a0 tx=0x7f91a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:56.084 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.083+0000 7f9197fff700 1 -- 172.21.15.152:0/886284045 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f91a00126f0 con 0x7f91b0060200 2023-12-06T04:00:56.291 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.289+0000 7f91b5b06700 1 -- 172.21.15.152:0/886284045 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f91b010a300 con 0x7f91b0060200 2023-12-06T04:00:56.291 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.289+0000 7f9197fff700 1 -- 172.21.15.152:0/886284045 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f91a0012330 con 0x7f91b0060200 2023-12-06T04:00:56.292 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:56.292 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:56.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.291+0000 7f91b5b06700 1 -- 172.21.15.152:0/886284045 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f919802ff60 msgr2=0x7f9198032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:56.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.291+0000 7f91b5b06700 1 --2- 172.21.15.152:0/886284045 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f919802ff60 0x7f9198032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f91a40096a0 tx=0x7f91a4006b40).stop 2023-12-06T04:00:56.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.291+0000 7f91b5b06700 1 -- 172.21.15.152:0/886284045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91b0060200 msgr2=0x7f91b01088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:56.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.291+0000 7f91b5b06700 1 --2- 172.21.15.152:0/886284045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91b0060200 0x7f91b01088d0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f91a0013510 tx=0x7f91a000b040).stop 2023-12-06T04:00:56.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.291+0000 7f91b5b06700 1 -- 172.21.15.152:0/886284045 shutdown_connections 2023-12-06T04:00:56.293 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.291+0000 7f91b5b06700 1 --2- 172.21.15.152:0/886284045 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f919802ff60 0x7f9198032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:56.294 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.291+0000 7f91b5b06700 1 --2- 172.21.15.152:0/886284045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f91b0060200 0x7f91b01088d0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:56.294 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.291+0000 7f91b5b06700 1 -- 172.21.15.152:0/886284045 >> 172.21.15.152:0/886284045 conn(0x7f91b00f0b70 msgr2=0x7f91b00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:56.294 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.291+0000 7f91b5b06700 1 -- 172.21.15.152:0/886284045 shutdown_connections 2023-12-06T04:00:56.294 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:56.291+0000 7f91b5b06700 1 -- 172.21.15.152:0/886284045 wait complete. 2023-12-06T04:00:56.294 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:00:57.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:56 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:56.291086+0000 mon.smithi132 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.152:0/886284045' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:00:57.677 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:00:57.678 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:00:58.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:57 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:56.462466+0000 mgr.smithi132.gokxja (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:00:58.764 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.762+0000 7f150bb83700 1 -- 172.21.15.152:0/3517913713 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15040f5410 msgr2=0x7f15040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:58.764 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.762+0000 7f150bb83700 1 --2- 172.21.15.152:0/3517913713 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15040f5410 0x7f15040f5830 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f14f4009fb0 tx=0x7f14f4009b30).stop 2023-12-06T04:00:58.764 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.762+0000 7f150bb83700 1 -- 172.21.15.152:0/3517913713 shutdown_connections 2023-12-06T04:00:58.764 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.762+0000 7f150bb83700 1 --2- 172.21.15.152:0/3517913713 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15040f5410 0x7f15040f5830 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:58.765 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.762+0000 7f150bb83700 1 -- 172.21.15.152:0/3517913713 >> 172.21.15.152:0/3517913713 conn(0x7f15040f0b70 msgr2=0x7f15040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:58.765 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.762+0000 7f150bb83700 1 -- 172.21.15.152:0/3517913713 shutdown_connections 2023-12-06T04:00:58.765 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.762+0000 7f150bb83700 1 -- 172.21.15.152:0/3517913713 wait complete. 2023-12-06T04:00:58.766 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.763+0000 7f150bb83700 1 Processor -- start 2023-12-06T04:00:58.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.763+0000 7f150bb83700 1 -- start start 2023-12-06T04:00:58.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.763+0000 7f150bb83700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15040f5410 0x7f1504109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:58.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.763+0000 7f150bb83700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1504060220 con 0x7f15040f5410 2023-12-06T04:00:58.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.763+0000 7f150991f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15040f5410 0x7f1504109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:58.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.763+0000 7f150991f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15040f5410 0x7f1504109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:34592/0 (socket says 172.21.15.152:34592) 2023-12-06T04:00:58.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.763+0000 7f150991f700 1 -- 172.21.15.152:0/1022214027 learned_addr learned my addr 172.21.15.152:0/1022214027 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:00:58.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.764+0000 7f150991f700 1 -- 172.21.15.152:0/1022214027 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14f4005040 con 0x7f15040f5410 2023-12-06T04:00:58.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.764+0000 7f150991f700 1 --2- 172.21.15.152:0/1022214027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15040f5410 0x7f1504109450 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f14f4013510 tx=0x7f14f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:58.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.764+0000 7f14fa7fc700 1 -- 172.21.15.152:0/1022214027 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14f4009320 con 0x7f15040f5410 2023-12-06T04:00:58.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.764+0000 7f150bb83700 1 -- 172.21.15.152:0/1022214027 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1504060420 con 0x7f15040f5410 2023-12-06T04:00:58.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.764+0000 7f14fa7fc700 1 -- 172.21.15.152:0/1022214027 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f14f4013f00 con 0x7f15040f5410 2023-12-06T04:00:58.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.764+0000 7f14fa7fc700 1 -- 172.21.15.152:0/1022214027 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14f4014d90 con 0x7f15040f5410 2023-12-06T04:00:58.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.764+0000 7f150bb83700 1 -- 172.21.15.152:0/1022214027 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1504109b00 con 0x7f15040f5410 2023-12-06T04:00:58.767 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.765+0000 7f14fa7fc700 1 -- 172.21.15.152:0/1022214027 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f14f4014420 con 0x7f15040f5410 2023-12-06T04:00:58.768 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.765+0000 7f150bb83700 1 -- 172.21.15.152:0/1022214027 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1504059100 con 0x7f15040f5410 2023-12-06T04:00:58.768 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.765+0000 7f14fa7fc700 1 --2- 172.21.15.152:0/1022214027 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f14f002ff60 0x7f14f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:00:58.768 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.765+0000 7f14fa7fc700 1 -- 172.21.15.152:0/1022214027 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14f4041110 con 0x7f15040f5410 2023-12-06T04:00:58.768 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.765+0000 7f150911e700 1 --2- 172.21.15.152:0/1022214027 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f14f002ff60 0x7f14f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:00:58.768 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.765+0000 7f150911e700 1 --2- 172.21.15.152:0/1022214027 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f14f002ff60 0x7f14f0032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f15000096a0 tx=0x7f1500006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:00:58.771 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.770+0000 7f14fa7fc700 1 -- 172.21.15.152:0/1022214027 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14f40095f0 con 0x7f15040f5410 2023-12-06T04:00:58.979 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.977+0000 7f150bb83700 1 -- 172.21.15.152:0/1022214027 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1504045ad0 con 0x7f15040f5410 2023-12-06T04:00:58.979 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.978+0000 7f14fa7fc700 1 -- 172.21.15.152:0/1022214027 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f14f401db70 con 0x7f15040f5410 2023-12-06T04:00:58.980 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:00:58.980 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:00:58.982 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.979+0000 7f150bb83700 1 -- 172.21.15.152:0/1022214027 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f14f002ff60 msgr2=0x7f14f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:58.982 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.979+0000 7f150bb83700 1 --2- 172.21.15.152:0/1022214027 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f14f002ff60 0x7f14f0032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f15000096a0 tx=0x7f1500006b40).stop 2023-12-06T04:00:58.982 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.979+0000 7f150bb83700 1 -- 172.21.15.152:0/1022214027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15040f5410 msgr2=0x7f1504109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:00:58.982 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.979+0000 7f150bb83700 1 --2- 172.21.15.152:0/1022214027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15040f5410 0x7f1504109450 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f14f4013510 tx=0x7f14f400b040).stop 2023-12-06T04:00:58.982 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.980+0000 7f150bb83700 1 -- 172.21.15.152:0/1022214027 shutdown_connections 2023-12-06T04:00:58.982 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.980+0000 7f150bb83700 1 --2- 172.21.15.152:0/1022214027 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f14f002ff60 0x7f14f0032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:58.982 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.980+0000 7f150bb83700 1 --2- 172.21.15.152:0/1022214027 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15040f5410 0x7f1504109450 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:00:58.982 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.980+0000 7f150bb83700 1 -- 172.21.15.152:0/1022214027 >> 172.21.15.152:0/1022214027 conn(0x7f15040f0b70 msgr2=0x7f15040f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:00:58.983 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.980+0000 7f150bb83700 1 -- 172.21.15.152:0/1022214027 shutdown_connections 2023-12-06T04:00:58.983 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:00:58.980+0000 7f150bb83700 1 -- 172.21.15.152:0/1022214027 wait complete. 2023-12-06T04:00:58.983 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:00.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:59 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:00:58.462824+0000 mgr.smithi132.gokxja (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:00.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:00:59 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:00:58.979266+0000 mon.smithi132 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.152:0/1022214027' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:00.365 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:00.365 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:01.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.495+0000 7f9e9dd68700 1 -- 172.21.15.152:0/162560343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e980f6510 msgr2=0x7f9e980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:01.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.495+0000 7f9e9dd68700 1 --2- 172.21.15.152:0/162560343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e980f6510 0x7f9e980f6930 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f9e90009960 tx=0x7f9e90004d10).stop 2023-12-06T04:01:01.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.495+0000 7f9e9dd68700 1 -- 172.21.15.152:0/162560343 shutdown_connections 2023-12-06T04:01:01.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.495+0000 7f9e9dd68700 1 --2- 172.21.15.152:0/162560343 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e980f6510 0x7f9e980f6930 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:01.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.495+0000 7f9e9dd68700 1 -- 172.21.15.152:0/162560343 >> 172.21.15.152:0/162560343 conn(0x7f9e980f1c70 msgr2=0x7f9e980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:01.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.495+0000 7f9e9dd68700 1 -- 172.21.15.152:0/162560343 shutdown_connections 2023-12-06T04:01:01.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.495+0000 7f9e9dd68700 1 -- 172.21.15.152:0/162560343 wait complete. 2023-12-06T04:01:01.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.495+0000 7f9e9dd68700 1 Processor -- start 2023-12-06T04:01:01.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.496+0000 7f9e9dd68700 1 -- start start 2023-12-06T04:01:01.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.496+0000 7f9e9dd68700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e980f6510 0x7f9e9806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:01.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.496+0000 7f9e9dd68700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e98069e20 con 0x7f9e980f6510 2023-12-06T04:01:01.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.496+0000 7f9e977fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e980f6510 0x7f9e9806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:01.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.496+0000 7f9e977fe700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e980f6510 0x7f9e9806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:33350/0 (socket says 172.21.15.152:33350) 2023-12-06T04:01:01.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.496+0000 7f9e977fe700 1 -- 172.21.15.152:0/3372888299 learned_addr learned my addr 172.21.15.152:0/3372888299 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:01.500 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.497+0000 7f9e977fe700 1 -- 172.21.15.152:0/3372888299 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e90009640 con 0x7f9e980f6510 2023-12-06T04:01:01.500 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.497+0000 7f9e977fe700 1 --2- 172.21.15.152:0/3372888299 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e980f6510 0x7f9e9806b9a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f9e90004b70 tx=0x7f9e90004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:01.500 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.497+0000 7f9e7ffff700 1 -- 172.21.15.152:0/3372888299 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e90018070 con 0x7f9e980f6510 2023-12-06T04:01:01.501 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.497+0000 7f9e7ffff700 1 -- 172.21.15.152:0/3372888299 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e9000b870 con 0x7f9e980f6510 2023-12-06T04:01:01.501 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.497+0000 7f9e9dd68700 1 -- 172.21.15.152:0/3372888299 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e98069fc0 con 0x7f9e980f6510 2023-12-06T04:01:01.501 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.497+0000 7f9e7ffff700 1 -- 172.21.15.152:0/3372888299 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e9001c560 con 0x7f9e980f6510 2023-12-06T04:01:01.501 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.497+0000 7f9e9dd68700 1 -- 172.21.15.152:0/3372888299 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e9806a450 con 0x7f9e980f6510 2023-12-06T04:01:01.501 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.498+0000 7f9e7ffff700 1 -- 172.21.15.152:0/3372888299 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e90013630 con 0x7f9e980f6510 2023-12-06T04:01:01.501 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.498+0000 7f9e9dd68700 1 -- 172.21.15.152:0/3372888299 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e98059100 con 0x7f9e980f6510 2023-12-06T04:01:01.501 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.498+0000 7f9e7ffff700 1 --2- 172.21.15.152:0/3372888299 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9e8002ff60 0x7f9e80032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:01.501 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.498+0000 7f9e7ffff700 1 -- 172.21.15.152:0/3372888299 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e90041290 con 0x7f9e980f6510 2023-12-06T04:01:01.501 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.498+0000 7f9e96ffd700 1 --2- 172.21.15.152:0/3372888299 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9e8002ff60 0x7f9e80032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:01.501 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.499+0000 7f9e96ffd700 1 --2- 172.21.15.152:0/3372888299 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9e8002ff60 0x7f9e80032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f9e8c0096a0 tx=0x7f9e8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:01.504 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.503+0000 7f9e7ffff700 1 -- 172.21.15.152:0/3372888299 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e90012770 con 0x7f9e980f6510 2023-12-06T04:01:01.711 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.709+0000 7f9e9dd68700 1 -- 172.21.15.152:0/3372888299 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e980fbf60 con 0x7f9e980f6510 2023-12-06T04:01:01.712 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.710+0000 7f9e7ffff700 1 -- 172.21.15.152:0/3372888299 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9e90012380 con 0x7f9e980f6510 2023-12-06T04:01:01.712 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:01.712 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:01.713 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.712+0000 7f9e9dd68700 1 -- 172.21.15.152:0/3372888299 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9e8002ff60 msgr2=0x7f9e80032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:01.713 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.712+0000 7f9e9dd68700 1 --2- 172.21.15.152:0/3372888299 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9e8002ff60 0x7f9e80032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f9e8c0096a0 tx=0x7f9e8c006b40).stop 2023-12-06T04:01:01.713 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.712+0000 7f9e9dd68700 1 -- 172.21.15.152:0/3372888299 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e980f6510 msgr2=0x7f9e9806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:01.714 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.712+0000 7f9e9dd68700 1 --2- 172.21.15.152:0/3372888299 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e980f6510 0x7f9e9806b9a0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f9e90004b70 tx=0x7f9e90004d10).stop 2023-12-06T04:01:01.714 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.712+0000 7f9e9dd68700 1 -- 172.21.15.152:0/3372888299 shutdown_connections 2023-12-06T04:01:01.714 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.712+0000 7f9e9dd68700 1 --2- 172.21.15.152:0/3372888299 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f9e8002ff60 0x7f9e80032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:01.714 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.712+0000 7f9e9dd68700 1 --2- 172.21.15.152:0/3372888299 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f9e980f6510 0x7f9e9806b9a0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:01.714 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.712+0000 7f9e9dd68700 1 -- 172.21.15.152:0/3372888299 >> 172.21.15.152:0/3372888299 conn(0x7f9e980f1c70 msgr2=0x7f9e980f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:01.714 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.712+0000 7f9e9dd68700 1 -- 172.21.15.152:0/3372888299 shutdown_connections 2023-12-06T04:01:01.714 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:01.712+0000 7f9e9dd68700 1 -- 172.21.15.152:0/3372888299 wait complete. 2023-12-06T04:01:01.714 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:02.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:01 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:00.463171+0000 mgr.smithi132.gokxja (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:03.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:02 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:01.711737+0000 mon.smithi132 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.152:0/3372888299' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:03.118 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:03.118 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:04.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:03 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:02.463398+0000 mgr.smithi132.gokxja (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:04.200 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.195+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2787924208 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e5c060280 msgr2=0x7f5e5c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:04.200 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.195+0000 7f5e60a7d700 1 --2- 172.21.15.152:0/2787924208 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e5c060280 0x7f5e5c0f90f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f5e4c009fb0 tx=0x7f5e4c009b30).stop 2023-12-06T04:01:04.200 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.195+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2787924208 shutdown_connections 2023-12-06T04:01:04.200 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.195+0000 7f5e60a7d700 1 --2- 172.21.15.152:0/2787924208 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e5c060280 0x7f5e5c0f90f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:04.200 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.195+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2787924208 >> 172.21.15.152:0/2787924208 conn(0x7f5e5c0f0b70 msgr2=0x7f5e5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:04.200 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.195+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2787924208 shutdown_connections 2023-12-06T04:01:04.200 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.195+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2787924208 wait complete. 2023-12-06T04:01:04.200 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.196+0000 7f5e60a7d700 1 Processor -- start 2023-12-06T04:01:04.200 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.196+0000 7f5e60a7d700 1 -- start start 2023-12-06T04:01:04.200 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.196+0000 7f5e60a7d700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e5c060280 0x7f5e5c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:04.200 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.196+0000 7f5e60a7d700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e5c106bd0 con 0x7f5e5c060280 2023-12-06T04:01:04.200 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.196+0000 7f5e5a59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e5c060280 0x7f5e5c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:04.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.196+0000 7f5e5a59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e5c060280 0x7f5e5c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:33358/0 (socket says 172.21.15.152:33358) 2023-12-06T04:01:04.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.196+0000 7f5e5a59c700 1 -- 172.21.15.152:0/2377464124 learned_addr learned my addr 172.21.15.152:0/2377464124 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:04.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.197+0000 7f5e5a59c700 1 -- 172.21.15.152:0/2377464124 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4c005040 con 0x7f5e5c060280 2023-12-06T04:01:04.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.197+0000 7f5e5a59c700 1 --2- 172.21.15.152:0/2377464124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e5c060280 0x7f5e5c1066e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f5e4c013510 tx=0x7f5e4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:04.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.197+0000 7f5e52ffd700 1 -- 172.21.15.152:0/2377464124 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e4c009320 con 0x7f5e5c060280 2023-12-06T04:01:04.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.197+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2377464124 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e5c106dd0 con 0x7f5e5c060280 2023-12-06T04:01:04.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.197+0000 7f5e52ffd700 1 -- 172.21.15.152:0/2377464124 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e4c013f00 con 0x7f5e5c060280 2023-12-06T04:01:04.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.197+0000 7f5e52ffd700 1 -- 172.21.15.152:0/2377464124 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e4c014dc0 con 0x7f5e5c060280 2023-12-06T04:01:04.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.197+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2377464124 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e5c107260 con 0x7f5e5c060280 2023-12-06T04:01:04.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.198+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2377464124 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e5c045ad0 con 0x7f5e5c060280 2023-12-06T04:01:04.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.198+0000 7f5e52ffd700 1 -- 172.21.15.152:0/2377464124 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e4c014420 con 0x7f5e5c060280 2023-12-06T04:01:04.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.198+0000 7f5e52ffd700 1 --2- 172.21.15.152:0/2377464124 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5e4802ff60 0x7f5e48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:04.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.198+0000 7f5e52ffd700 1 -- 172.21.15.152:0/2377464124 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e4c041050 con 0x7f5e5c060280 2023-12-06T04:01:04.201 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.198+0000 7f5e59d9b700 1 --2- 172.21.15.152:0/2377464124 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5e4802ff60 0x7f5e48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:04.202 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.198+0000 7f5e59d9b700 1 --2- 172.21.15.152:0/2377464124 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5e4802ff60 0x7f5e48032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f5e440096a0 tx=0x7f5e44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:04.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.203+0000 7f5e52ffd700 1 -- 172.21.15.152:0/2377464124 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e4c0095f0 con 0x7f5e5c060280 2023-12-06T04:01:04.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.409+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2377464124 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5e5c108110 con 0x7f5e5c060280 2023-12-06T04:01:04.411 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.410+0000 7f5e52ffd700 1 -- 172.21.15.152:0/2377464124 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5e4c01cb10 con 0x7f5e5c060280 2023-12-06T04:01:04.412 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:04.412 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:04.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.412+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2377464124 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5e4802ff60 msgr2=0x7f5e48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:04.413 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.412+0000 7f5e60a7d700 1 --2- 172.21.15.152:0/2377464124 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5e4802ff60 0x7f5e48032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f5e440096a0 tx=0x7f5e44006b40).stop 2023-12-06T04:01:04.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.412+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2377464124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e5c060280 msgr2=0x7f5e5c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:04.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.412+0000 7f5e60a7d700 1 --2- 172.21.15.152:0/2377464124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e5c060280 0x7f5e5c1066e0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f5e4c013510 tx=0x7f5e4c00b040).stop 2023-12-06T04:01:04.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.412+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2377464124 shutdown_connections 2023-12-06T04:01:04.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.412+0000 7f5e60a7d700 1 --2- 172.21.15.152:0/2377464124 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5e4802ff60 0x7f5e48032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:04.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.412+0000 7f5e60a7d700 1 --2- 172.21.15.152:0/2377464124 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5e5c060280 0x7f5e5c1066e0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:04.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.412+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2377464124 >> 172.21.15.152:0/2377464124 conn(0x7f5e5c0f0b70 msgr2=0x7f5e5c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:04.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.412+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2377464124 shutdown_connections 2023-12-06T04:01:04.414 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:04.412+0000 7f5e60a7d700 1 -- 172.21.15.152:0/2377464124 wait complete. 2023-12-06T04:01:04.415 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:05.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:04 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:04.411562+0000 mon.smithi132 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.152:0/2377464124' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:05.784 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:05.784 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:06.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:05 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:04.463661+0000 mgr.smithi132.gokxja (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:06.901 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.899+0000 7fbf7eb09700 1 -- 172.21.15.152:0/1092107486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf780f5410 msgr2=0x7fbf780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:06.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.899+0000 7fbf7eb09700 1 --2- 172.21.15.152:0/1092107486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf780f5410 0x7fbf780f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fbf6c009fb0 tx=0x7fbf6c009b30).stop 2023-12-06T04:01:06.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.899+0000 7fbf7eb09700 1 -- 172.21.15.152:0/1092107486 shutdown_connections 2023-12-06T04:01:06.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.899+0000 7fbf7eb09700 1 --2- 172.21.15.152:0/1092107486 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf780f5410 0x7fbf780f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:06.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.899+0000 7fbf7eb09700 1 -- 172.21.15.152:0/1092107486 >> 172.21.15.152:0/1092107486 conn(0x7fbf780f0b70 msgr2=0x7fbf780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:06.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.899+0000 7fbf7eb09700 1 -- 172.21.15.152:0/1092107486 shutdown_connections 2023-12-06T04:01:06.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.899+0000 7fbf7eb09700 1 -- 172.21.15.152:0/1092107486 wait complete. 2023-12-06T04:01:06.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.900+0000 7fbf7eb09700 1 Processor -- start 2023-12-06T04:01:06.902 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.900+0000 7fbf7eb09700 1 -- start start 2023-12-06T04:01:06.904 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.900+0000 7fbf7eb09700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf780f5410 0x7fbf78105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:06.904 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.900+0000 7fbf7eb09700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf78105b20 con 0x7fbf780f5410 2023-12-06T04:01:06.904 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.900+0000 7fbf7c8a5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf780f5410 0x7fbf78105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:06.904 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.900+0000 7fbf7c8a5700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf780f5410 0x7fbf78105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:33378/0 (socket says 172.21.15.152:33378) 2023-12-06T04:01:06.904 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.900+0000 7fbf7c8a5700 1 -- 172.21.15.152:0/541516424 learned_addr learned my addr 172.21.15.152:0/541516424 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:06.904 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.901+0000 7fbf7c8a5700 1 -- 172.21.15.152:0/541516424 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf6c005040 con 0x7fbf780f5410 2023-12-06T04:01:06.904 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.901+0000 7fbf7c8a5700 1 --2- 172.21.15.152:0/541516424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf780f5410 0x7fbf78105630 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fbf6c014000 tx=0x7fbf6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:06.904 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.901+0000 7fbf757fa700 1 -- 172.21.15.152:0/541516424 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf6c0095f0 con 0x7fbf780f5410 2023-12-06T04:01:06.904 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.901+0000 7fbf7eb09700 1 -- 172.21.15.152:0/541516424 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf78105d20 con 0x7fbf780f5410 2023-12-06T04:01:06.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.901+0000 7fbf757fa700 1 -- 172.21.15.152:0/541516424 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf6c007e70 con 0x7fbf780f5410 2023-12-06T04:01:06.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.901+0000 7fbf757fa700 1 -- 172.21.15.152:0/541516424 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf6c014ea0 con 0x7fbf780f5410 2023-12-06T04:01:06.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.901+0000 7fbf7eb09700 1 -- 172.21.15.152:0/541516424 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf781061b0 con 0x7fbf780f5410 2023-12-06T04:01:06.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.902+0000 7fbf757fa700 1 -- 172.21.15.152:0/541516424 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbf6c025070 con 0x7fbf780f5410 2023-12-06T04:01:06.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.902+0000 7fbf7eb09700 1 -- 172.21.15.152:0/541516424 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf78045ad0 con 0x7fbf780f5410 2023-12-06T04:01:06.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.902+0000 7fbf757fa700 1 --2- 172.21.15.152:0/541516424 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbf6802ff60 0x7fbf68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:06.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.902+0000 7fbf757fa700 1 -- 172.21.15.152:0/541516424 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbf6c040a30 con 0x7fbf780f5410 2023-12-06T04:01:06.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.902+0000 7fbf77fff700 1 --2- 172.21.15.152:0/541516424 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbf6802ff60 0x7fbf68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:06.905 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.902+0000 7fbf77fff700 1 --2- 172.21.15.152:0/541516424 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbf6802ff60 0x7fbf68032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fbf640096a0 tx=0x7fbf64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:06.908 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:06.907+0000 7fbf757fa700 1 -- 172.21.15.152:0/541516424 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf6c0079c0 con 0x7fbf780f5410 2023-12-06T04:01:07.116 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:07.114+0000 7fbf7eb09700 1 -- 172.21.15.152:0/541516424 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf780fae60 con 0x7fbf780f5410 2023-12-06T04:01:07.116 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:07.115+0000 7fbf757fa700 1 -- 172.21.15.152:0/541516424 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbf6c00f300 con 0x7fbf780f5410 2023-12-06T04:01:07.117 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:07.117 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:07.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:07.116+0000 7fbf7eb09700 1 -- 172.21.15.152:0/541516424 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbf6802ff60 msgr2=0x7fbf68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:07.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:07.116+0000 7fbf7eb09700 1 --2- 172.21.15.152:0/541516424 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbf6802ff60 0x7fbf68032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fbf640096a0 tx=0x7fbf64006b40).stop 2023-12-06T04:01:07.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:07.116+0000 7fbf7eb09700 1 -- 172.21.15.152:0/541516424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf780f5410 msgr2=0x7fbf78105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:07.118 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:07.116+0000 7fbf7eb09700 1 --2- 172.21.15.152:0/541516424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf780f5410 0x7fbf78105630 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fbf6c014000 tx=0x7fbf6c00b040).stop 2023-12-06T04:01:07.119 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:07.117+0000 7fbf7eb09700 1 -- 172.21.15.152:0/541516424 shutdown_connections 2023-12-06T04:01:07.119 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:07.117+0000 7fbf7eb09700 1 --2- 172.21.15.152:0/541516424 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fbf6802ff60 0x7fbf68032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:07.119 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:07.117+0000 7fbf7eb09700 1 --2- 172.21.15.152:0/541516424 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fbf780f5410 0x7fbf78105630 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:07.119 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:07.117+0000 7fbf7eb09700 1 -- 172.21.15.152:0/541516424 >> 172.21.15.152:0/541516424 conn(0x7fbf780f0b70 msgr2=0x7fbf780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:07.119 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:07.117+0000 7fbf7eb09700 1 -- 172.21.15.152:0/541516424 shutdown_connections 2023-12-06T04:01:07.119 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:07.117+0000 7fbf7eb09700 1 -- 172.21.15.152:0/541516424 wait complete. 2023-12-06T04:01:07.119 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:08.109 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:07 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:06.463978+0000 mgr.smithi132.gokxja (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:08.110 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:07 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:07.116369+0000 mon.smithi132 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.152:0/541516424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:08.499 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:08.499 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:09.675 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.672+0000 7f37aaedf700 1 -- 172.21.15.152:0/884310332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37a40f5410 msgr2=0x7f37a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:09.675 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.672+0000 7f37aaedf700 1 --2- 172.21.15.152:0/884310332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37a40f5410 0x7f37a40f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f3798009fb0 tx=0x7f3798009b30).stop 2023-12-06T04:01:09.675 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.672+0000 7f37aaedf700 1 -- 172.21.15.152:0/884310332 shutdown_connections 2023-12-06T04:01:09.675 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.672+0000 7f37aaedf700 1 --2- 172.21.15.152:0/884310332 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37a40f5410 0x7f37a40f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:09.675 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.672+0000 7f37aaedf700 1 -- 172.21.15.152:0/884310332 >> 172.21.15.152:0/884310332 conn(0x7f37a40f0b70 msgr2=0x7f37a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:09.675 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.672+0000 7f37aaedf700 1 -- 172.21.15.152:0/884310332 shutdown_connections 2023-12-06T04:01:09.676 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.673+0000 7f37aaedf700 1 -- 172.21.15.152:0/884310332 wait complete. 2023-12-06T04:01:09.676 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.673+0000 7f37aaedf700 1 Processor -- start 2023-12-06T04:01:09.676 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.673+0000 7f37aaedf700 1 -- start start 2023-12-06T04:01:09.677 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.674+0000 7f37aaedf700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37a40f5410 0x7f37a4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:09.677 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.674+0000 7f37aaedf700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a4060240 con 0x7f37a40f5410 2023-12-06T04:01:09.677 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.674+0000 7f37a8c7b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37a40f5410 0x7f37a4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:09.677 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.674+0000 7f37a8c7b700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37a40f5410 0x7f37a4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:39494/0 (socket says 172.21.15.152:39494) 2023-12-06T04:01:09.677 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.674+0000 7f37a8c7b700 1 -- 172.21.15.152:0/3335536451 learned_addr learned my addr 172.21.15.152:0/3335536451 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:09.678 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.674+0000 7f37a8c7b700 1 -- 172.21.15.152:0/3335536451 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3798005040 con 0x7f37a40f5410 2023-12-06T04:01:09.678 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.674+0000 7f37a8c7b700 1 --2- 172.21.15.152:0/3335536451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37a40f5410 0x7f37a4109430 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f3798013510 tx=0x7f379800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:09.678 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.674+0000 7f37a17fa700 1 -- 172.21.15.152:0/3335536451 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3798009320 con 0x7f37a40f5410 2023-12-06T04:01:09.678 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.674+0000 7f37aaedf700 1 -- 172.21.15.152:0/3335536451 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a4060440 con 0x7f37a40f5410 2023-12-06T04:01:09.678 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.674+0000 7f37a17fa700 1 -- 172.21.15.152:0/3335536451 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3798013f00 con 0x7f37a40f5410 2023-12-06T04:01:09.678 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.674+0000 7f37a17fa700 1 -- 172.21.15.152:0/3335536451 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3798014d90 con 0x7f37a40f5410 2023-12-06T04:01:09.678 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.674+0000 7f37aaedf700 1 -- 172.21.15.152:0/3335536451 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a4109ae0 con 0x7f37a40f5410 2023-12-06T04:01:09.678 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.675+0000 7f37a17fa700 1 -- 172.21.15.152:0/3335536451 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37980095f0 con 0x7f37a40f5410 2023-12-06T04:01:09.678 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.675+0000 7f37aaedf700 1 -- 172.21.15.152:0/3335536451 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a4045ad0 con 0x7f37a40f5410 2023-12-06T04:01:09.678 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.675+0000 7f37a17fa700 1 --2- 172.21.15.152:0/3335536451 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f379402ff60 0x7f3794032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:09.678 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.675+0000 7f37a17fa700 1 -- 172.21.15.152:0/3335536451 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37980410c0 con 0x7f37a40f5410 2023-12-06T04:01:09.678 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.676+0000 7f37a3fff700 1 --2- 172.21.15.152:0/3335536451 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f379402ff60 0x7f3794032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:09.679 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.676+0000 7f37a3fff700 1 --2- 172.21.15.152:0/3335536451 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f379402ff60 0x7f3794032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f37900096a0 tx=0x7f3790006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:09.682 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.680+0000 7f37a17fa700 1 -- 172.21.15.152:0/3335536451 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f379801d7f0 con 0x7f37a40f5410 2023-12-06T04:01:09.886 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.884+0000 7f37aaedf700 1 -- 172.21.15.152:0/3335536451 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37a40fae60 con 0x7f37a40f5410 2023-12-06T04:01:09.887 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.885+0000 7f37a17fa700 1 -- 172.21.15.152:0/3335536451 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3798010230 con 0x7f37a40f5410 2023-12-06T04:01:09.887 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:09.887 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:09.888 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.886+0000 7f37aaedf700 1 -- 172.21.15.152:0/3335536451 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f379402ff60 msgr2=0x7f3794032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:09.888 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.886+0000 7f37aaedf700 1 --2- 172.21.15.152:0/3335536451 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f379402ff60 0x7f3794032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f37900096a0 tx=0x7f3790006b40).stop 2023-12-06T04:01:09.888 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.886+0000 7f37aaedf700 1 -- 172.21.15.152:0/3335536451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37a40f5410 msgr2=0x7f37a4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:09.888 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.886+0000 7f37aaedf700 1 --2- 172.21.15.152:0/3335536451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37a40f5410 0x7f37a4109430 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f3798013510 tx=0x7f379800b040).stop 2023-12-06T04:01:09.889 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.887+0000 7f37aaedf700 1 -- 172.21.15.152:0/3335536451 shutdown_connections 2023-12-06T04:01:09.889 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.887+0000 7f37aaedf700 1 --2- 172.21.15.152:0/3335536451 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f379402ff60 0x7f3794032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:09.889 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.887+0000 7f37aaedf700 1 --2- 172.21.15.152:0/3335536451 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f37a40f5410 0x7f37a4109430 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:09.889 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.887+0000 7f37aaedf700 1 -- 172.21.15.152:0/3335536451 >> 172.21.15.152:0/3335536451 conn(0x7f37a40f0b70 msgr2=0x7f37a40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:09.889 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.887+0000 7f37aaedf700 1 -- 172.21.15.152:0/3335536451 shutdown_connections 2023-12-06T04:01:09.889 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:09.887+0000 7f37aaedf700 1 -- 172.21.15.152:0/3335536451 wait complete. 2023-12-06T04:01:09.889 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:10.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:09 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:08.464365+0000 mgr.smithi132.gokxja (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:11.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:10 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:09.886607+0000 mon.smithi132 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.152:0/3335536451' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:11.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:11.270 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:12.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:11 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:10.464661+0000 mgr.smithi132.gokxja (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:12.406 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.403+0000 7f2294f2f700 1 -- 172.21.15.152:0/2169664051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2290060220 msgr2=0x7f22900f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:12.406 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.403+0000 7f2294f2f700 1 --2- 172.21.15.152:0/2169664051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2290060220 0x7f22900f9150 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f2280009fb0 tx=0x7f2280009b30).stop 2023-12-06T04:01:12.406 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.403+0000 7f2294f2f700 1 -- 172.21.15.152:0/2169664051 shutdown_connections 2023-12-06T04:01:12.406 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.403+0000 7f2294f2f700 1 --2- 172.21.15.152:0/2169664051 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2290060220 0x7f22900f9150 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:12.406 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.403+0000 7f2294f2f700 1 -- 172.21.15.152:0/2169664051 >> 172.21.15.152:0/2169664051 conn(0x7f22900f0b70 msgr2=0x7f22900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:12.406 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.403+0000 7f2294f2f700 1 -- 172.21.15.152:0/2169664051 shutdown_connections 2023-12-06T04:01:12.406 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.403+0000 7f2294f2f700 1 -- 172.21.15.152:0/2169664051 wait complete. 2023-12-06T04:01:12.406 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.404+0000 7f2294f2f700 1 Processor -- start 2023-12-06T04:01:12.406 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.404+0000 7f2294f2f700 1 -- start start 2023-12-06T04:01:12.406 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.404+0000 7f2294f2f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2290060220 0x7f22900f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:12.406 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.404+0000 7f2294f2f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22900f5050 con 0x7f2290060220 2023-12-06T04:01:12.407 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.404+0000 7f228e59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2290060220 0x7f22900f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:12.407 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.404+0000 7f228e59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2290060220 0x7f22900f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:39506/0 (socket says 172.21.15.152:39506) 2023-12-06T04:01:12.407 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.404+0000 7f228e59c700 1 -- 172.21.15.152:0/3720217993 learned_addr learned my addr 172.21.15.152:0/3720217993 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:12.408 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.404+0000 7f228e59c700 1 -- 172.21.15.152:0/3720217993 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2280005040 con 0x7f2290060220 2023-12-06T04:01:12.408 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.404+0000 7f228e59c700 1 --2- 172.21.15.152:0/3720217993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2290060220 0x7f22900f6bd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f2280013510 tx=0x7f228000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:12.408 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.405+0000 7f2286ffd700 1 -- 172.21.15.152:0/3720217993 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2280009320 con 0x7f2290060220 2023-12-06T04:01:12.408 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.405+0000 7f2294f2f700 1 -- 172.21.15.152:0/3720217993 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22900f51f0 con 0x7f2290060220 2023-12-06T04:01:12.408 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.405+0000 7f2286ffd700 1 -- 172.21.15.152:0/3720217993 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2280013f00 con 0x7f2290060220 2023-12-06T04:01:12.408 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.405+0000 7f2286ffd700 1 -- 172.21.15.152:0/3720217993 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2280014dc0 con 0x7f2290060220 2023-12-06T04:01:12.408 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.405+0000 7f2294f2f700 1 -- 172.21.15.152:0/3720217993 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22900f5680 con 0x7f2290060220 2023-12-06T04:01:12.409 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.406+0000 7f2294f2f700 1 -- 172.21.15.152:0/3720217993 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2290059100 con 0x7f2290060220 2023-12-06T04:01:12.409 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.406+0000 7f2286ffd700 1 -- 172.21.15.152:0/3720217993 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f22800095f0 con 0x7f2290060220 2023-12-06T04:01:12.409 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.406+0000 7f2286ffd700 1 --2- 172.21.15.152:0/3720217993 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f227c02ff60 0x7f227c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:12.409 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.406+0000 7f2286ffd700 1 -- 172.21.15.152:0/3720217993 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2280041020 con 0x7f2290060220 2023-12-06T04:01:12.409 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.406+0000 7f228dd9b700 1 --2- 172.21.15.152:0/3720217993 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f227c02ff60 0x7f227c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:12.409 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.406+0000 7f228dd9b700 1 --2- 172.21.15.152:0/3720217993 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f227c02ff60 0x7f227c032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f22780096a0 tx=0x7f2278006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:12.412 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.410+0000 7f2286ffd700 1 -- 172.21.15.152:0/3720217993 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2280014420 con 0x7f2290060220 2023-12-06T04:01:12.617 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.615+0000 7f2294f2f700 1 -- 172.21.15.152:0/3720217993 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2290045ad0 con 0x7f2290060220 2023-12-06T04:01:12.618 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.616+0000 7f2286ffd700 1 -- 172.21.15.152:0/3720217993 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f228001c920 con 0x7f2290060220 2023-12-06T04:01:12.618 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:12.618 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:12.620 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.618+0000 7f2294f2f700 1 -- 172.21.15.152:0/3720217993 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f227c02ff60 msgr2=0x7f227c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:12.620 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.618+0000 7f2294f2f700 1 --2- 172.21.15.152:0/3720217993 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f227c02ff60 0x7f227c032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f22780096a0 tx=0x7f2278006b40).stop 2023-12-06T04:01:12.620 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.618+0000 7f2294f2f700 1 -- 172.21.15.152:0/3720217993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2290060220 msgr2=0x7f22900f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:12.620 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.618+0000 7f2294f2f700 1 --2- 172.21.15.152:0/3720217993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2290060220 0x7f22900f6bd0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f2280013510 tx=0x7f228000b040).stop 2023-12-06T04:01:12.620 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.618+0000 7f2294f2f700 1 -- 172.21.15.152:0/3720217993 shutdown_connections 2023-12-06T04:01:12.620 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.618+0000 7f2294f2f700 1 --2- 172.21.15.152:0/3720217993 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f227c02ff60 0x7f227c032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:12.620 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.618+0000 7f2294f2f700 1 --2- 172.21.15.152:0/3720217993 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f2290060220 0x7f22900f6bd0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:12.620 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.618+0000 7f2294f2f700 1 -- 172.21.15.152:0/3720217993 >> 172.21.15.152:0/3720217993 conn(0x7f22900f0b70 msgr2=0x7f22900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:12.620 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.618+0000 7f2294f2f700 1 -- 172.21.15.152:0/3720217993 shutdown_connections 2023-12-06T04:01:12.620 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:12.618+0000 7f2294f2f700 1 -- 172.21.15.152:0/3720217993 wait complete. 2023-12-06T04:01:12.621 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:13.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:12 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:12.617718+0000 mon.smithi132 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.152:0/3720217993' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:14.024 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:14.024 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:14.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:13 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:12.464864+0000 mgr.smithi132.gokxja (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:15.207 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.205+0000 7f5803fb8700 1 -- 172.21.15.152:0/964887139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57fc0f5410 msgr2=0x7f57fc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:15.207 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.205+0000 7f5803fb8700 1 --2- 172.21.15.152:0/964887139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57fc0f5410 0x7f57fc0f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f57f8009fb0 tx=0x7f57f8009b30).stop 2023-12-06T04:01:15.207 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.205+0000 7f5803fb8700 1 -- 172.21.15.152:0/964887139 shutdown_connections 2023-12-06T04:01:15.207 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.205+0000 7f5803fb8700 1 --2- 172.21.15.152:0/964887139 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57fc0f5410 0x7f57fc0f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:15.207 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.205+0000 7f5803fb8700 1 -- 172.21.15.152:0/964887139 >> 172.21.15.152:0/964887139 conn(0x7f57fc0f0b70 msgr2=0x7f57fc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:15.207 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.205+0000 7f5803fb8700 1 -- 172.21.15.152:0/964887139 shutdown_connections 2023-12-06T04:01:15.207 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.205+0000 7f5803fb8700 1 -- 172.21.15.152:0/964887139 wait complete. 2023-12-06T04:01:15.210 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.206+0000 7f5803fb8700 1 Processor -- start 2023-12-06T04:01:15.210 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.206+0000 7f5803fb8700 1 -- start start 2023-12-06T04:01:15.210 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.206+0000 7f5803fb8700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57fc0f5410 0x7f57fc0fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:15.210 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.206+0000 7f5803fb8700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57fc0fb820 con 0x7f57fc0f5410 2023-12-06T04:01:15.210 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.206+0000 7f5801d54700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57fc0f5410 0x7f57fc0fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:15.210 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.206+0000 7f5801d54700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57fc0f5410 0x7f57fc0fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:39512/0 (socket says 172.21.15.152:39512) 2023-12-06T04:01:15.210 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.206+0000 7f5801d54700 1 -- 172.21.15.152:0/928220634 learned_addr learned my addr 172.21.15.152:0/928220634 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:15.210 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.206+0000 7f5801d54700 1 -- 172.21.15.152:0/928220634 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f8005040 con 0x7f57fc0f5410 2023-12-06T04:01:15.210 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.206+0000 7f5801d54700 1 --2- 172.21.15.152:0/928220634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57fc0f5410 0x7f57fc0fd3a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f57f8014000 tx=0x7f57f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:15.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.207+0000 7f57f27fc700 1 -- 172.21.15.152:0/928220634 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57f80095f0 con 0x7f57fc0f5410 2023-12-06T04:01:15.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.207+0000 7f5803fb8700 1 -- 172.21.15.152:0/928220634 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57fc0fb9c0 con 0x7f57fc0f5410 2023-12-06T04:01:15.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.207+0000 7f57f27fc700 1 -- 172.21.15.152:0/928220634 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57f8007e70 con 0x7f57fc0f5410 2023-12-06T04:01:15.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.207+0000 7f57f27fc700 1 -- 172.21.15.152:0/928220634 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57f8014ea0 con 0x7f57fc0f5410 2023-12-06T04:01:15.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.207+0000 7f5803fb8700 1 -- 172.21.15.152:0/928220634 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57fc0fbe50 con 0x7f57fc0f5410 2023-12-06T04:01:15.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.207+0000 7f57f27fc700 1 -- 172.21.15.152:0/928220634 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57f8025070 con 0x7f57fc0f5410 2023-12-06T04:01:15.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.208+0000 7f57f27fc700 1 --2- 172.21.15.152:0/928220634 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f57e802ff60 0x7f57e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:15.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.208+0000 7f57f27fc700 1 -- 172.21.15.152:0/928220634 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57f8040d20 con 0x7f57fc0f5410 2023-12-06T04:01:15.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.208+0000 7f5803fb8700 1 -- 172.21.15.152:0/928220634 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57fc059100 con 0x7f57fc0f5410 2023-12-06T04:01:15.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.208+0000 7f5801553700 1 --2- 172.21.15.152:0/928220634 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f57e802ff60 0x7f57e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:15.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.208+0000 7f5801553700 1 --2- 172.21.15.152:0/928220634 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f57e802ff60 0x7f57e8032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f57ec0096a0 tx=0x7f57ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:15.214 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.212+0000 7f57f27fc700 1 -- 172.21.15.152:0/928220634 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57f8026350 con 0x7f57fc0f5410 2023-12-06T04:01:15.419 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.418+0000 7f5803fb8700 1 -- 172.21.15.152:0/928220634 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57fc0fc550 con 0x7f57fc0f5410 2023-12-06T04:01:15.420 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.418+0000 7f57f27fc700 1 -- 172.21.15.152:0/928220634 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f57f8012330 con 0x7f57fc0f5410 2023-12-06T04:01:15.420 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:15.420 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:15.421 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.420+0000 7f5803fb8700 1 -- 172.21.15.152:0/928220634 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f57e802ff60 msgr2=0x7f57e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:15.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.420+0000 7f5803fb8700 1 --2- 172.21.15.152:0/928220634 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f57e802ff60 0x7f57e8032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f57ec0096a0 tx=0x7f57ec006b40).stop 2023-12-06T04:01:15.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.420+0000 7f5803fb8700 1 -- 172.21.15.152:0/928220634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57fc0f5410 msgr2=0x7f57fc0fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:15.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.420+0000 7f5803fb8700 1 --2- 172.21.15.152:0/928220634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57fc0f5410 0x7f57fc0fd3a0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f57f8014000 tx=0x7f57f800b040).stop 2023-12-06T04:01:15.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.420+0000 7f5803fb8700 1 -- 172.21.15.152:0/928220634 shutdown_connections 2023-12-06T04:01:15.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.420+0000 7f5803fb8700 1 --2- 172.21.15.152:0/928220634 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f57e802ff60 0x7f57e8032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:15.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.420+0000 7f5803fb8700 1 --2- 172.21.15.152:0/928220634 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f57fc0f5410 0x7f57fc0fd3a0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:15.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.420+0000 7f5803fb8700 1 -- 172.21.15.152:0/928220634 >> 172.21.15.152:0/928220634 conn(0x7f57fc0f0b70 msgr2=0x7f57fc05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:15.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.420+0000 7f5803fb8700 1 -- 172.21.15.152:0/928220634 shutdown_connections 2023-12-06T04:01:15.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:15.420+0000 7f5803fb8700 1 -- 172.21.15.152:0/928220634 wait complete. 2023-12-06T04:01:15.422 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:16.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:15 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:14.465177+0000 mgr.smithi132.gokxja (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:16.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:15 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:15.419916+0000 mon.smithi132 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.152:0/928220634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:16.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:16.803 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:17.980 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.977+0000 7fb05c434700 1 -- 172.21.15.152:0/3085003379 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb054060280 msgr2=0x7fb0540f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:17.981 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.977+0000 7fb05c434700 1 --2- 172.21.15.152:0/3085003379 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb054060280 0x7fb0540f90f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fb050009fb0 tx=0x7fb050009b30).stop 2023-12-06T04:01:17.981 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.977+0000 7fb05c434700 1 -- 172.21.15.152:0/3085003379 shutdown_connections 2023-12-06T04:01:17.981 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.977+0000 7fb05c434700 1 --2- 172.21.15.152:0/3085003379 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb054060280 0x7fb0540f90f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:17.981 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.977+0000 7fb05c434700 1 -- 172.21.15.152:0/3085003379 >> 172.21.15.152:0/3085003379 conn(0x7fb0540f0b70 msgr2=0x7fb0540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:17.981 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.977+0000 7fb05c434700 1 -- 172.21.15.152:0/3085003379 shutdown_connections 2023-12-06T04:01:17.981 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.977+0000 7fb05c434700 1 -- 172.21.15.152:0/3085003379 wait complete. 2023-12-06T04:01:17.981 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.977+0000 7fb05c434700 1 Processor -- start 2023-12-06T04:01:17.981 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.977+0000 7fb05c434700 1 -- start start 2023-12-06T04:01:17.981 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.978+0000 7fb05c434700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb054060280 0x7fb0540f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:17.981 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.978+0000 7fb05c434700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0540f4ff0 con 0x7fb054060280 2023-12-06T04:01:17.982 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.978+0000 7fb05a1d0700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb054060280 0x7fb0540f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:17.982 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.978+0000 7fb05a1d0700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb054060280 0x7fb0540f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:39528/0 (socket says 172.21.15.152:39528) 2023-12-06T04:01:17.982 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.978+0000 7fb05a1d0700 1 -- 172.21.15.152:0/155426392 learned_addr learned my addr 172.21.15.152:0/155426392 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:17.982 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.978+0000 7fb05a1d0700 1 -- 172.21.15.152:0/155426392 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb050005040 con 0x7fb054060280 2023-12-06T04:01:17.982 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.978+0000 7fb05a1d0700 1 --2- 172.21.15.152:0/155426392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb054060280 0x7fb0540f6b70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fb050013510 tx=0x7fb05000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:17.982 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.978+0000 7fb04affd700 1 -- 172.21.15.152:0/155426392 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb050009320 con 0x7fb054060280 2023-12-06T04:01:17.983 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.978+0000 7fb05c434700 1 -- 172.21.15.152:0/155426392 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0540f5190 con 0x7fb054060280 2023-12-06T04:01:17.983 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.978+0000 7fb04affd700 1 -- 172.21.15.152:0/155426392 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb050013f00 con 0x7fb054060280 2023-12-06T04:01:17.983 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.978+0000 7fb04affd700 1 -- 172.21.15.152:0/155426392 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb050014dc0 con 0x7fb054060280 2023-12-06T04:01:17.984 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.979+0000 7fb05c434700 1 -- 172.21.15.152:0/155426392 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0540f5620 con 0x7fb054060280 2023-12-06T04:01:17.984 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.979+0000 7fb04affd700 1 -- 172.21.15.152:0/155426392 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0500095f0 con 0x7fb054060280 2023-12-06T04:01:17.984 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.979+0000 7fb05c434700 1 -- 172.21.15.152:0/155426392 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb054059100 con 0x7fb054060280 2023-12-06T04:01:17.984 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.979+0000 7fb04affd700 1 --2- 172.21.15.152:0/155426392 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb04002ff60 0x7fb040032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:17.984 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.979+0000 7fb04affd700 1 -- 172.21.15.152:0/155426392 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb050041000 con 0x7fb054060280 2023-12-06T04:01:17.984 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.980+0000 7fb0599cf700 1 --2- 172.21.15.152:0/155426392 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb04002ff60 0x7fb040032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:17.984 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.980+0000 7fb0599cf700 1 --2- 172.21.15.152:0/155426392 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb04002ff60 0x7fb040032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fb0440096a0 tx=0x7fb044006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:17.986 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:17.984+0000 7fb04affd700 1 -- 172.21.15.152:0/155426392 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb0500268c0 con 0x7fb054060280 2023-12-06T04:01:18.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:17 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:16.465530+0000 mgr.smithi132.gokxja (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:18.191 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:18.189+0000 7fb05c434700 1 -- 172.21.15.152:0/155426392 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0540fbf50 con 0x7fb054060280 2023-12-06T04:01:18.191 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:18.189+0000 7fb04affd700 1 -- 172.21.15.152:0/155426392 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb050010230 con 0x7fb054060280 2023-12-06T04:01:18.191 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:18.191 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:18.193 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:18.191+0000 7fb05c434700 1 -- 172.21.15.152:0/155426392 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb04002ff60 msgr2=0x7fb040032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:18.193 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:18.191+0000 7fb05c434700 1 --2- 172.21.15.152:0/155426392 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb04002ff60 0x7fb040032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fb0440096a0 tx=0x7fb044006b40).stop 2023-12-06T04:01:18.194 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:18.191+0000 7fb05c434700 1 -- 172.21.15.152:0/155426392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb054060280 msgr2=0x7fb0540f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:18.194 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:18.191+0000 7fb05c434700 1 --2- 172.21.15.152:0/155426392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb054060280 0x7fb0540f6b70 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fb050013510 tx=0x7fb05000b040).stop 2023-12-06T04:01:18.194 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:18.191+0000 7fb05c434700 1 -- 172.21.15.152:0/155426392 shutdown_connections 2023-12-06T04:01:18.194 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:18.191+0000 7fb05c434700 1 --2- 172.21.15.152:0/155426392 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fb04002ff60 0x7fb040032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:18.194 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:18.191+0000 7fb05c434700 1 --2- 172.21.15.152:0/155426392 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fb054060280 0x7fb0540f6b70 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:18.194 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:18.191+0000 7fb05c434700 1 -- 172.21.15.152:0/155426392 >> 172.21.15.152:0/155426392 conn(0x7fb0540f0b70 msgr2=0x7fb0540f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:18.194 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:18.191+0000 7fb05c434700 1 -- 172.21.15.152:0/155426392 shutdown_connections 2023-12-06T04:01:18.194 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:18.191+0000 7fb05c434700 1 -- 172.21.15.152:0/155426392 wait complete. 2023-12-06T04:01:18.194 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:19.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:18 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:18.191088+0000 mon.smithi132 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.152:0/155426392' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:19.564 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:19.565 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:20.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:19 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:18.465825+0000 mgr.smithi132.gokxja (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:20.631 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.629+0000 7fa8672c9700 1 -- 172.21.15.152:0/1768138359 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa860060280 msgr2=0x7fa8600f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:20.631 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.629+0000 7fa8672c9700 1 --2- 172.21.15.152:0/1768138359 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa860060280 0x7fa8600f90f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fa85c009fb0 tx=0x7fa85c009b30).stop 2023-12-06T04:01:20.631 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.629+0000 7fa8672c9700 1 -- 172.21.15.152:0/1768138359 shutdown_connections 2023-12-06T04:01:20.631 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.629+0000 7fa8672c9700 1 --2- 172.21.15.152:0/1768138359 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa860060280 0x7fa8600f90f0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:20.632 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.629+0000 7fa8672c9700 1 -- 172.21.15.152:0/1768138359 >> 172.21.15.152:0/1768138359 conn(0x7fa8600f0b70 msgr2=0x7fa8600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:20.632 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.629+0000 7fa8672c9700 1 -- 172.21.15.152:0/1768138359 shutdown_connections 2023-12-06T04:01:20.632 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.629+0000 7fa8672c9700 1 -- 172.21.15.152:0/1768138359 wait complete. 2023-12-06T04:01:20.632 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.630+0000 7fa8672c9700 1 Processor -- start 2023-12-06T04:01:20.632 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.630+0000 7fa8672c9700 1 -- start start 2023-12-06T04:01:20.633 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.630+0000 7fa8672c9700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa860060280 0x7fa8601066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:20.633 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.630+0000 7fa8672c9700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa860106bd0 con 0x7fa860060280 2023-12-06T04:01:20.633 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.630+0000 7fa865065700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa860060280 0x7fa8601066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:20.633 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.630+0000 7fa865065700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa860060280 0x7fa8601066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:41980/0 (socket says 172.21.15.152:41980) 2023-12-06T04:01:20.633 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.630+0000 7fa865065700 1 -- 172.21.15.152:0/2369775045 learned_addr learned my addr 172.21.15.152:0/2369775045 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:20.633 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.630+0000 7fa865065700 1 -- 172.21.15.152:0/2369775045 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c005040 con 0x7fa860060280 2023-12-06T04:01:20.633 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.631+0000 7fa865065700 1 --2- 172.21.15.152:0/2369775045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa860060280 0x7fa8601066e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fa85c013510 tx=0x7fa85c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:20.634 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.631+0000 7fa855ffb700 1 -- 172.21.15.152:0/2369775045 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa85c009320 con 0x7fa860060280 2023-12-06T04:01:20.634 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.631+0000 7fa855ffb700 1 -- 172.21.15.152:0/2369775045 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa85c013f00 con 0x7fa860060280 2023-12-06T04:01:20.634 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.631+0000 7fa8672c9700 1 -- 172.21.15.152:0/2369775045 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa860106dd0 con 0x7fa860060280 2023-12-06T04:01:20.634 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.631+0000 7fa855ffb700 1 -- 172.21.15.152:0/2369775045 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa85c014dc0 con 0x7fa860060280 2023-12-06T04:01:20.635 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.631+0000 7fa8672c9700 1 -- 172.21.15.152:0/2369775045 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa860107260 con 0x7fa860060280 2023-12-06T04:01:20.635 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.632+0000 7fa8672c9700 1 -- 172.21.15.152:0/2369775045 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa860045ad0 con 0x7fa860060280 2023-12-06T04:01:20.635 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.632+0000 7fa855ffb700 1 -- 172.21.15.152:0/2369775045 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa85c0095f0 con 0x7fa860060280 2023-12-06T04:01:20.635 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.632+0000 7fa855ffb700 1 --2- 172.21.15.152:0/2369775045 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa84c02ff60 0x7fa84c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:20.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.632+0000 7fa855ffb700 1 -- 172.21.15.152:0/2369775045 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa85c041040 con 0x7fa860060280 2023-12-06T04:01:20.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.632+0000 7fa864864700 1 --2- 172.21.15.152:0/2369775045 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa84c02ff60 0x7fa84c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:20.636 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.633+0000 7fa864864700 1 --2- 172.21.15.152:0/2369775045 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa84c02ff60 0x7fa84c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fa8500096a0 tx=0x7fa850006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:20.638 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.637+0000 7fa855ffb700 1 -- 172.21.15.152:0/2369775045 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa85c014420 con 0x7fa860060280 2023-12-06T04:01:20.847 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.845+0000 7fa8672c9700 1 -- 172.21.15.152:0/2369775045 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa860108110 con 0x7fa860060280 2023-12-06T04:01:20.847 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.846+0000 7fa855ffb700 1 -- 172.21.15.152:0/2369775045 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa85c010230 con 0x7fa860060280 2023-12-06T04:01:20.847 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:20.848 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:20.849 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.848+0000 7fa8672c9700 1 -- 172.21.15.152:0/2369775045 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa84c02ff60 msgr2=0x7fa84c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:20.849 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.848+0000 7fa8672c9700 1 --2- 172.21.15.152:0/2369775045 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa84c02ff60 0x7fa84c032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fa8500096a0 tx=0x7fa850006b40).stop 2023-12-06T04:01:20.850 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.848+0000 7fa8672c9700 1 -- 172.21.15.152:0/2369775045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa860060280 msgr2=0x7fa8601066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:20.850 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.848+0000 7fa8672c9700 1 --2- 172.21.15.152:0/2369775045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa860060280 0x7fa8601066e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fa85c013510 tx=0x7fa85c00b040).stop 2023-12-06T04:01:20.850 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.848+0000 7fa8672c9700 1 -- 172.21.15.152:0/2369775045 shutdown_connections 2023-12-06T04:01:20.850 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.848+0000 7fa8672c9700 1 --2- 172.21.15.152:0/2369775045 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7fa84c02ff60 0x7fa84c032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:20.850 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.848+0000 7fa8672c9700 1 --2- 172.21.15.152:0/2369775045 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7fa860060280 0x7fa8601066e0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:20.850 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.848+0000 7fa8672c9700 1 -- 172.21.15.152:0/2369775045 >> 172.21.15.152:0/2369775045 conn(0x7fa8600f0b70 msgr2=0x7fa8600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:20.850 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.848+0000 7fa8672c9700 1 -- 172.21.15.152:0/2369775045 shutdown_connections 2023-12-06T04:01:20.850 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:20.848+0000 7fa8672c9700 1 -- 172.21.15.152:0/2369775045 wait complete. 2023-12-06T04:01:20.850 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:22.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:20.466177+0000 mgr.smithi132.gokxja (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:22.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:21 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:20.847276+0000 mon.smithi132 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.152:0/2369775045' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:22.243 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:22.244 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:23.371 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.369+0000 7f8eabd5c700 1 -- 172.21.15.152:0/791985666 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ea40f5950 msgr2=0x7f8ea40f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:23.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.369+0000 7f8eabd5c700 1 --2- 172.21.15.152:0/791985666 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ea40f5950 0x7f8ea40f5d70 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f8e94009fb0 tx=0x7f8e94009b30).stop 2023-12-06T04:01:23.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.369+0000 7f8eabd5c700 1 -- 172.21.15.152:0/791985666 shutdown_connections 2023-12-06T04:01:23.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.369+0000 7f8eabd5c700 1 --2- 172.21.15.152:0/791985666 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ea40f5950 0x7f8ea40f5d70 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:23.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.369+0000 7f8eabd5c700 1 -- 172.21.15.152:0/791985666 >> 172.21.15.152:0/791985666 conn(0x7f8ea40f0260 msgr2=0x7f8ea40f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:23.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.369+0000 7f8eabd5c700 1 -- 172.21.15.152:0/791985666 shutdown_connections 2023-12-06T04:01:23.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.369+0000 7f8eabd5c700 1 -- 172.21.15.152:0/791985666 wait complete. 2023-12-06T04:01:23.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.370+0000 7f8eabd5c700 1 Processor -- start 2023-12-06T04:01:23.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.370+0000 7f8eabd5c700 1 -- start start 2023-12-06T04:01:23.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.370+0000 7f8eabd5c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ea40f5950 0x7f8ea4109410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:23.372 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.370+0000 7f8eabd5c700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea40f5330 con 0x7f8ea40f5950 2023-12-06T04:01:23.374 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.370+0000 7f8ea9af8700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ea40f5950 0x7f8ea4109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:23.374 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.371+0000 7f8ea9af8700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ea40f5950 0x7f8ea4109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:41996/0 (socket says 172.21.15.152:41996) 2023-12-06T04:01:23.374 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.371+0000 7f8ea9af8700 1 -- 172.21.15.152:0/1617706876 learned_addr learned my addr 172.21.15.152:0/1617706876 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:23.374 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.371+0000 7f8ea9af8700 1 -- 172.21.15.152:0/1617706876 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e94005040 con 0x7f8ea40f5950 2023-12-06T04:01:23.374 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.371+0000 7f8ea9af8700 1 --2- 172.21.15.152:0/1617706876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ea40f5950 0x7f8ea4109410 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f8e94014000 tx=0x7f8e9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:23.374 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.371+0000 7f8e9a7fc700 1 -- 172.21.15.152:0/1617706876 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e940095f0 con 0x7f8ea40f5950 2023-12-06T04:01:23.374 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.371+0000 7f8eabd5c700 1 -- 172.21.15.152:0/1617706876 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea40f5530 con 0x7f8ea40f5950 2023-12-06T04:01:23.374 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.371+0000 7f8e9a7fc700 1 -- 172.21.15.152:0/1617706876 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e94007e70 con 0x7f8ea40f5950 2023-12-06T04:01:23.374 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.371+0000 7f8e9a7fc700 1 -- 172.21.15.152:0/1617706876 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e94014ea0 con 0x7f8ea40f5950 2023-12-06T04:01:23.374 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.371+0000 7f8eabd5c700 1 -- 172.21.15.152:0/1617706876 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea4109ac0 con 0x7f8ea40f5950 2023-12-06T04:01:23.374 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.372+0000 7f8e9a7fc700 1 -- 172.21.15.152:0/1617706876 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8e94025070 con 0x7f8ea40f5950 2023-12-06T04:01:23.374 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.372+0000 7f8e9a7fc700 1 --2- 172.21.15.152:0/1617706876 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8e9002ff60 0x7f8e90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:23.375 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.372+0000 7f8eabd5c700 1 -- 172.21.15.152:0/1617706876 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea4045ad0 con 0x7f8ea40f5950 2023-12-06T04:01:23.375 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.372+0000 7f8e9a7fc700 1 -- 172.21.15.152:0/1617706876 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8e94040930 con 0x7f8ea40f5950 2023-12-06T04:01:23.375 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.372+0000 7f8ea92f7700 1 --2- 172.21.15.152:0/1617706876 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8e9002ff60 0x7f8e90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:23.375 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.373+0000 7f8ea92f7700 1 --2- 172.21.15.152:0/1617706876 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8e9002ff60 0x7f8e90032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f8ea00096a0 tx=0x7f8ea0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:23.378 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.377+0000 7f8e9a7fc700 1 -- 172.21.15.152:0/1617706876 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e9400f4e0 con 0x7f8ea40f5950 2023-12-06T04:01:23.585 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.583+0000 7f8eabd5c700 1 -- 172.21.15.152:0/1617706876 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ea410a2e0 con 0x7f8ea40f5950 2023-12-06T04:01:23.585 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.584+0000 7f8e9a7fc700 1 -- 172.21.15.152:0/1617706876 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8e9400f300 con 0x7f8ea40f5950 2023-12-06T04:01:23.585 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:23.586 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:23.587 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.586+0000 7f8eabd5c700 1 -- 172.21.15.152:0/1617706876 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8e9002ff60 msgr2=0x7f8e90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:23.587 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.586+0000 7f8eabd5c700 1 --2- 172.21.15.152:0/1617706876 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8e9002ff60 0x7f8e90032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f8ea00096a0 tx=0x7f8ea0006b40).stop 2023-12-06T04:01:23.587 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.586+0000 7f8eabd5c700 1 -- 172.21.15.152:0/1617706876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ea40f5950 msgr2=0x7f8ea4109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:23.587 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.586+0000 7f8eabd5c700 1 --2- 172.21.15.152:0/1617706876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ea40f5950 0x7f8ea4109410 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f8e94014000 tx=0x7f8e9400b040).stop 2023-12-06T04:01:23.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.586+0000 7f8eabd5c700 1 -- 172.21.15.152:0/1617706876 shutdown_connections 2023-12-06T04:01:23.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.586+0000 7f8eabd5c700 1 --2- 172.21.15.152:0/1617706876 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f8e9002ff60 0x7f8e90032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:23.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.586+0000 7f8eabd5c700 1 --2- 172.21.15.152:0/1617706876 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f8ea40f5950 0x7f8ea4109410 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:23.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.586+0000 7f8eabd5c700 1 -- 172.21.15.152:0/1617706876 >> 172.21.15.152:0/1617706876 conn(0x7f8ea40f0260 msgr2=0x7f8ea40f0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:23.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.586+0000 7f8eabd5c700 1 -- 172.21.15.152:0/1617706876 shutdown_connections 2023-12-06T04:01:23.588 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:23.586+0000 7f8eabd5c700 1 -- 172.21.15.152:0/1617706876 wait complete. 2023-12-06T04:01:23.588 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:24.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:22.466576+0000 mgr.smithi132.gokxja (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:24.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:23 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:23.585382+0000 mon.smithi132 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.152:0/1617706876' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:24.921 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:24.922 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:25.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.991+0000 7f1b5c929700 1 -- 172.21.15.152:0/150455473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b580f5410 msgr2=0x7f1b580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:25.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.991+0000 7f1b5c929700 1 --2- 172.21.15.152:0/150455473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b580f5410 0x7f1b580f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f1b4c009fb0 tx=0x7f1b4c009b30).stop 2023-12-06T04:01:25.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.991+0000 7f1b5c929700 1 -- 172.21.15.152:0/150455473 shutdown_connections 2023-12-06T04:01:25.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.991+0000 7f1b5c929700 1 --2- 172.21.15.152:0/150455473 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b580f5410 0x7f1b580f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:25.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.991+0000 7f1b5c929700 1 -- 172.21.15.152:0/150455473 >> 172.21.15.152:0/150455473 conn(0x7f1b580f0b70 msgr2=0x7f1b580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:25.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.991+0000 7f1b5c929700 1 -- 172.21.15.152:0/150455473 shutdown_connections 2023-12-06T04:01:25.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.991+0000 7f1b5c929700 1 -- 172.21.15.152:0/150455473 wait complete. 2023-12-06T04:01:25.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.992+0000 7f1b5c929700 1 Processor -- start 2023-12-06T04:01:25.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.992+0000 7f1b5c929700 1 -- start start 2023-12-06T04:01:25.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.992+0000 7f1b5c929700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b580f5410 0x7f1b58105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:25.995 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.992+0000 7f1b5c929700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b58105b20 con 0x7f1b580f5410 2023-12-06T04:01:25.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.993+0000 7f1b5659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b580f5410 0x7f1b58105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:25.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.993+0000 7f1b5659c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b580f5410 0x7f1b58105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:42012/0 (socket says 172.21.15.152:42012) 2023-12-06T04:01:25.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.993+0000 7f1b5659c700 1 -- 172.21.15.152:0/851618580 learned_addr learned my addr 172.21.15.152:0/851618580 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:25.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.993+0000 7f1b5659c700 1 -- 172.21.15.152:0/851618580 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4c005040 con 0x7f1b580f5410 2023-12-06T04:01:25.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.993+0000 7f1b5659c700 1 --2- 172.21.15.152:0/851618580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b580f5410 0x7f1b58105630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f1b4c013510 tx=0x7f1b4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:25.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.993+0000 7f1b46ffd700 1 -- 172.21.15.152:0/851618580 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b4c009320 con 0x7f1b580f5410 2023-12-06T04:01:25.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.993+0000 7f1b5c929700 1 -- 172.21.15.152:0/851618580 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b58105d20 con 0x7f1b580f5410 2023-12-06T04:01:25.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.993+0000 7f1b46ffd700 1 -- 172.21.15.152:0/851618580 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1b4c013f00 con 0x7f1b580f5410 2023-12-06T04:01:25.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.993+0000 7f1b46ffd700 1 -- 172.21.15.152:0/851618580 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b4c014d90 con 0x7f1b580f5410 2023-12-06T04:01:25.996 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.993+0000 7f1b5c929700 1 -- 172.21.15.152:0/851618580 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b581061b0 con 0x7f1b580f5410 2023-12-06T04:01:25.997 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.994+0000 7f1b46ffd700 1 -- 172.21.15.152:0/851618580 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1b4c014420 con 0x7f1b580f5410 2023-12-06T04:01:25.997 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.994+0000 7f1b5c929700 1 -- 172.21.15.152:0/851618580 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b58045ad0 con 0x7f1b580f5410 2023-12-06T04:01:25.997 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.994+0000 7f1b46ffd700 1 --2- 172.21.15.152:0/851618580 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1b4002ff60 0x7f1b40032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:25.997 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.994+0000 7f1b46ffd700 1 -- 172.21.15.152:0/851618580 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1b4c041050 con 0x7f1b580f5410 2023-12-06T04:01:25.997 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.994+0000 7f1b55d9b700 1 --2- 172.21.15.152:0/851618580 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1b4002ff60 0x7f1b40032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:25.997 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.995+0000 7f1b55d9b700 1 --2- 172.21.15.152:0/851618580 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1b4002ff60 0x7f1b40032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f1b480096a0 tx=0x7f1b48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:26.000 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:25.999+0000 7f1b46ffd700 1 -- 172.21.15.152:0/851618580 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1b4c0095f0 con 0x7f1b580f5410 2023-12-06T04:01:26.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:25 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:24.466953+0000 mgr.smithi132.gokxja (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:26.208 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:26.206+0000 7f1b5c929700 1 -- 172.21.15.152:0/851618580 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b580fae60 con 0x7f1b580f5410 2023-12-06T04:01:26.208 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:26.207+0000 7f1b46ffd700 1 -- 172.21.15.152:0/851618580 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1b4c01db70 con 0x7f1b580f5410 2023-12-06T04:01:26.209 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:26.209 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:26.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:26.208+0000 7f1b5c929700 1 -- 172.21.15.152:0/851618580 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1b4002ff60 msgr2=0x7f1b40032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:26.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:26.208+0000 7f1b5c929700 1 --2- 172.21.15.152:0/851618580 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1b4002ff60 0x7f1b40032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f1b480096a0 tx=0x7f1b48006b40).stop 2023-12-06T04:01:26.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:26.208+0000 7f1b5c929700 1 -- 172.21.15.152:0/851618580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b580f5410 msgr2=0x7f1b58105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:26.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:26.208+0000 7f1b5c929700 1 --2- 172.21.15.152:0/851618580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b580f5410 0x7f1b58105630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f1b4c013510 tx=0x7f1b4c00b040).stop 2023-12-06T04:01:26.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:26.208+0000 7f1b5c929700 1 -- 172.21.15.152:0/851618580 shutdown_connections 2023-12-06T04:01:26.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:26.208+0000 7f1b5c929700 1 --2- 172.21.15.152:0/851618580 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f1b4002ff60 0x7f1b40032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:26.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:26.208+0000 7f1b5c929700 1 --2- 172.21.15.152:0/851618580 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f1b580f5410 0x7f1b58105630 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:26.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:26.209+0000 7f1b5c929700 1 -- 172.21.15.152:0/851618580 >> 172.21.15.152:0/851618580 conn(0x7f1b580f0b70 msgr2=0x7f1b580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:26.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:26.209+0000 7f1b5c929700 1 -- 172.21.15.152:0/851618580 shutdown_connections 2023-12-06T04:01:26.211 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:26.209+0000 7f1b5c929700 1 -- 172.21.15.152:0/851618580 wait complete. 2023-12-06T04:01:26.211 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:27.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:26 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:26.208384+0000 mon.smithi132 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.152:0/851618580' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:27.577 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:27.577 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:28.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:27 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:26.467294+0000 mgr.smithi132.gokxja (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:28.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.743+0000 7f159fd17700 1 -- 172.21.15.152:0/4171836428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15980f6510 msgr2=0x7f15980f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:28.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.743+0000 7f159fd17700 1 --2- 172.21.15.152:0/4171836428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15980f6510 0x7f15980f6930 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f1588009fb0 tx=0x7f1588009b30).stop 2023-12-06T04:01:28.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.743+0000 7f159fd17700 1 -- 172.21.15.152:0/4171836428 shutdown_connections 2023-12-06T04:01:28.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.743+0000 7f159fd17700 1 --2- 172.21.15.152:0/4171836428 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15980f6510 0x7f15980f6930 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:28.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.743+0000 7f159fd17700 1 -- 172.21.15.152:0/4171836428 >> 172.21.15.152:0/4171836428 conn(0x7f15980f1c70 msgr2=0x7f15980f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:28.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.743+0000 7f159fd17700 1 -- 172.21.15.152:0/4171836428 shutdown_connections 2023-12-06T04:01:28.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.744+0000 7f159fd17700 1 -- 172.21.15.152:0/4171836428 wait complete. 2023-12-06T04:01:28.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.744+0000 7f159fd17700 1 Processor -- start 2023-12-06T04:01:28.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.744+0000 7f159fd17700 1 -- start start 2023-12-06T04:01:28.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.744+0000 7f159fd17700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15980f6510 0x7f1598106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:28.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.744+0000 7f159fd17700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1598106c20 con 0x7f15980f6510 2023-12-06T04:01:28.747 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.744+0000 7f159dab3700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15980f6510 0x7f1598106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:28.748 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.744+0000 7f159dab3700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15980f6510 0x7f1598106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:42036/0 (socket says 172.21.15.152:42036) 2023-12-06T04:01:28.748 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.744+0000 7f159dab3700 1 -- 172.21.15.152:0/2270106204 learned_addr learned my addr 172.21.15.152:0/2270106204 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:28.748 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.745+0000 7f159dab3700 1 -- 172.21.15.152:0/2270106204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1588005040 con 0x7f15980f6510 2023-12-06T04:01:28.748 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.745+0000 7f159dab3700 1 --2- 172.21.15.152:0/2270106204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15980f6510 0x7f1598106730 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f1588013510 tx=0x7f158800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:28.748 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.745+0000 7f158e7fc700 1 -- 172.21.15.152:0/2270106204 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1588009320 con 0x7f15980f6510 2023-12-06T04:01:28.748 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.745+0000 7f159fd17700 1 -- 172.21.15.152:0/2270106204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1598106e20 con 0x7f15980f6510 2023-12-06T04:01:28.748 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.745+0000 7f158e7fc700 1 -- 172.21.15.152:0/2270106204 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1588013f00 con 0x7f15980f6510 2023-12-06T04:01:28.748 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.745+0000 7f158e7fc700 1 -- 172.21.15.152:0/2270106204 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1588014d90 con 0x7f15980f6510 2023-12-06T04:01:28.748 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.745+0000 7f159fd17700 1 -- 172.21.15.152:0/2270106204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15981072b0 con 0x7f15980f6510 2023-12-06T04:01:28.749 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.746+0000 7f158e7fc700 1 -- 172.21.15.152:0/2270106204 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15880095f0 con 0x7f15980f6510 2023-12-06T04:01:28.749 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.746+0000 7f159fd17700 1 -- 172.21.15.152:0/2270106204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1598045ad0 con 0x7f15980f6510 2023-12-06T04:01:28.749 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.746+0000 7f158e7fc700 1 --2- 172.21.15.152:0/2270106204 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f158402ff60 0x7f1584032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:28.749 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.746+0000 7f158e7fc700 1 -- 172.21.15.152:0/2270106204 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1588041000 con 0x7f15980f6510 2023-12-06T04:01:28.749 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.746+0000 7f159d2b2700 1 --2- 172.21.15.152:0/2270106204 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f158402ff60 0x7f1584032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:28.749 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.747+0000 7f159d2b2700 1 --2- 172.21.15.152:0/2270106204 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f158402ff60 0x7f1584032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f15940096a0 tx=0x7f1594006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:28.752 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.751+0000 7f158e7fc700 1 -- 172.21.15.152:0/2270106204 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1588014420 con 0x7f15980f6510 2023-12-06T04:01:28.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.957+0000 7f159fd17700 1 -- 172.21.15.152:0/2270106204 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15980fbf60 con 0x7f15980f6510 2023-12-06T04:01:28.959 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.958+0000 7f158e7fc700 1 -- 172.21.15.152:0/2270106204 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1588010230 con 0x7f15980f6510 2023-12-06T04:01:28.960 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:28.960 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:28.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.959+0000 7f159fd17700 1 -- 172.21.15.152:0/2270106204 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f158402ff60 msgr2=0x7f1584032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:28.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.959+0000 7f159fd17700 1 --2- 172.21.15.152:0/2270106204 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f158402ff60 0x7f1584032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f15940096a0 tx=0x7f1594006b40).stop 2023-12-06T04:01:28.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.959+0000 7f159fd17700 1 -- 172.21.15.152:0/2270106204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15980f6510 msgr2=0x7f1598106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:28.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.959+0000 7f159fd17700 1 --2- 172.21.15.152:0/2270106204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15980f6510 0x7f1598106730 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f1588013510 tx=0x7f158800b040).stop 2023-12-06T04:01:28.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.959+0000 7f159fd17700 1 -- 172.21.15.152:0/2270106204 shutdown_connections 2023-12-06T04:01:28.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.959+0000 7f159fd17700 1 --2- 172.21.15.152:0/2270106204 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f158402ff60 0x7f1584032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:28.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.959+0000 7f159fd17700 1 --2- 172.21.15.152:0/2270106204 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f15980f6510 0x7f1598106730 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:28.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.959+0000 7f159fd17700 1 -- 172.21.15.152:0/2270106204 >> 172.21.15.152:0/2270106204 conn(0x7f15980f1c70 msgr2=0x7f15980f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:28.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.959+0000 7f159fd17700 1 -- 172.21.15.152:0/2270106204 shutdown_connections 2023-12-06T04:01:28.961 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:28.959+0000 7f159fd17700 1 -- 172.21.15.152:0/2270106204 wait complete. 2023-12-06T04:01:28.962 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:30.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:28.467642+0000 mgr.smithi132.gokxja (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:30.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:29 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:28.959235+0000 mon.smithi132 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.152:0/2270106204' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:30.349 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:30.349 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:31.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.495+0000 7f5d7ffff700 1 -- 172.21.15.152:0/1636015470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d800f51a0 msgr2=0x7f5d800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:31.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.495+0000 7f5d7ffff700 1 --2- 172.21.15.152:0/1636015470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d800f51a0 0x7f5d800f55c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f5d68009fb0 tx=0x7f5d68009b30).stop 2023-12-06T04:01:31.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.495+0000 7f5d7ffff700 1 -- 172.21.15.152:0/1636015470 shutdown_connections 2023-12-06T04:01:31.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.495+0000 7f5d7ffff700 1 --2- 172.21.15.152:0/1636015470 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d800f51a0 0x7f5d800f55c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:31.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.495+0000 7f5d7ffff700 1 -- 172.21.15.152:0/1636015470 >> 172.21.15.152:0/1636015470 conn(0x7f5d800f09a0 msgr2=0x7f5d800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:31.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.495+0000 7f5d7ffff700 1 -- 172.21.15.152:0/1636015470 shutdown_connections 2023-12-06T04:01:31.497 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.495+0000 7f5d7ffff700 1 -- 172.21.15.152:0/1636015470 wait complete. 2023-12-06T04:01:31.498 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.495+0000 7f5d7ffff700 1 Processor -- start 2023-12-06T04:01:31.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.496+0000 7f5d7ffff700 1 -- start start 2023-12-06T04:01:31.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.496+0000 7f5d7ffff700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d800f51a0 0x7f5d800601b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:31.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.496+0000 7f5d7ffff700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d801091d0 con 0x7f5d800f51a0 2023-12-06T04:01:31.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.496+0000 7f5d7effd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d800f51a0 0x7f5d800601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:31.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.496+0000 7f5d7effd700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d800f51a0 0x7f5d800601b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:35228/0 (socket says 172.21.15.152:35228) 2023-12-06T04:01:31.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.496+0000 7f5d7effd700 1 -- 172.21.15.152:0/29528281 learned_addr learned my addr 172.21.15.152:0/29528281 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:31.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.496+0000 7f5d7effd700 1 -- 172.21.15.152:0/29528281 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d68005040 con 0x7f5d800f51a0 2023-12-06T04:01:31.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.496+0000 7f5d7effd700 1 --2- 172.21.15.152:0/29528281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d800f51a0 0x7f5d800601b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f5d68013510 tx=0x7f5d6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:31.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.496+0000 7f5d777fe700 1 -- 172.21.15.152:0/29528281 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d68009320 con 0x7f5d800f51a0 2023-12-06T04:01:31.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.497+0000 7f5d7ffff700 1 -- 172.21.15.152:0/29528281 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d80109310 con 0x7f5d800f51a0 2023-12-06T04:01:31.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.497+0000 7f5d777fe700 1 -- 172.21.15.152:0/29528281 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5d68013f00 con 0x7f5d800f51a0 2023-12-06T04:01:31.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.497+0000 7f5d777fe700 1 -- 172.21.15.152:0/29528281 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d68014dc0 con 0x7f5d800f51a0 2023-12-06T04:01:31.499 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.497+0000 7f5d7ffff700 1 -- 172.21.15.152:0/29528281 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d80109800 con 0x7f5d800f51a0 2023-12-06T04:01:31.500 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.497+0000 7f5d777fe700 1 -- 172.21.15.152:0/29528281 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5d68014420 con 0x7f5d800f51a0 2023-12-06T04:01:31.500 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.497+0000 7f5d7ffff700 1 -- 172.21.15.152:0/29528281 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d80045ad0 con 0x7f5d800f51a0 2023-12-06T04:01:31.500 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.497+0000 7f5d777fe700 1 --2- 172.21.15.152:0/29528281 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5d6c02ff60 0x7f5d6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:31.500 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.498+0000 7f5d777fe700 1 -- 172.21.15.152:0/29528281 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5d68041050 con 0x7f5d800f51a0 2023-12-06T04:01:31.501 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.498+0000 7f5d7e7fc700 1 --2- 172.21.15.152:0/29528281 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5d6c02ff60 0x7f5d6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:31.501 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.498+0000 7f5d7e7fc700 1 --2- 172.21.15.152:0/29528281 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5d6c02ff60 0x7f5d6c032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f5d700096a0 tx=0x7f5d70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:31.504 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.502+0000 7f5d777fe700 1 -- 172.21.15.152:0/29528281 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5d680095f0 con 0x7f5d800f51a0 2023-12-06T04:01:31.712 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.710+0000 7f5d7ffff700 1 -- 172.21.15.152:0/29528281 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d8010a030 con 0x7f5d800f51a0 2023-12-06T04:01:31.712 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.712+0000 7f5d777fe700 1 -- 172.21.15.152:0/29528281 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5d6801cb10 con 0x7f5d800f51a0 2023-12-06T04:01:31.713 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:31.713 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:31.714 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.712+0000 7f5d7ffff700 1 -- 172.21.15.152:0/29528281 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5d6c02ff60 msgr2=0x7f5d6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:31.714 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.712+0000 7f5d7ffff700 1 --2- 172.21.15.152:0/29528281 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5d6c02ff60 0x7f5d6c032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f5d700096a0 tx=0x7f5d70006b40).stop 2023-12-06T04:01:31.714 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.712+0000 7f5d7ffff700 1 -- 172.21.15.152:0/29528281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d800f51a0 msgr2=0x7f5d800601b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:31.714 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.712+0000 7f5d7ffff700 1 --2- 172.21.15.152:0/29528281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d800f51a0 0x7f5d800601b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f5d68013510 tx=0x7f5d6800b040).stop 2023-12-06T04:01:31.714 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.713+0000 7f5d7ffff700 1 -- 172.21.15.152:0/29528281 shutdown_connections 2023-12-06T04:01:31.714 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.713+0000 7f5d7ffff700 1 --2- 172.21.15.152:0/29528281 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5d6c02ff60 0x7f5d6c032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:31.715 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.713+0000 7f5d7ffff700 1 --2- 172.21.15.152:0/29528281 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5d800f51a0 0x7f5d800601b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:31.715 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.713+0000 7f5d7ffff700 1 -- 172.21.15.152:0/29528281 >> 172.21.15.152:0/29528281 conn(0x7f5d800f09a0 msgr2=0x7f5d800f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:31.715 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.713+0000 7f5d7ffff700 1 -- 172.21.15.152:0/29528281 shutdown_connections 2023-12-06T04:01:31.715 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:31.713+0000 7f5d7ffff700 1 -- 172.21.15.152:0/29528281 wait complete. 2023-12-06T04:01:31.715 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:32.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:31 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:30.467991+0000 mgr.smithi132.gokxja (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:33.097 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:33.097 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:33.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:32 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:31.712461+0000 mon.smithi132 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.152:0/29528281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:34.107 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:33 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:32.468277+0000 mgr.smithi132.gokxja (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:34.202 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.200+0000 7f1030d04700 1 -- 172.21.15.152:0/1707341751 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f102c0f5410 msgr2=0x7f102c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:34.202 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.200+0000 7f1030d04700 1 --2- 172.21.15.152:0/1707341751 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f102c0f5410 0x7f102c0f5830 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f101c009fb0 tx=0x7f101c009b30).stop 2023-12-06T04:01:34.202 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.200+0000 7f1030d04700 1 -- 172.21.15.152:0/1707341751 shutdown_connections 2023-12-06T04:01:34.202 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.200+0000 7f1030d04700 1 --2- 172.21.15.152:0/1707341751 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f102c0f5410 0x7f102c0f5830 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:34.202 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.200+0000 7f1030d04700 1 -- 172.21.15.152:0/1707341751 >> 172.21.15.152:0/1707341751 conn(0x7f102c0f0b70 msgr2=0x7f102c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:34.202 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.200+0000 7f1030d04700 1 -- 172.21.15.152:0/1707341751 shutdown_connections 2023-12-06T04:01:34.203 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.200+0000 7f1030d04700 1 -- 172.21.15.152:0/1707341751 wait complete. 2023-12-06T04:01:34.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.201+0000 7f1030d04700 1 Processor -- start 2023-12-06T04:01:34.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.201+0000 7f1030d04700 1 -- start start 2023-12-06T04:01:34.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.201+0000 7f1030d04700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f102c0f5410 0x7f102c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:34.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.201+0000 7f1030d04700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f102c105b20 con 0x7f102c0f5410 2023-12-06T04:01:34.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.201+0000 7f102a59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f102c0f5410 0x7f102c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:34.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.201+0000 7f102a59c700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f102c0f5410 0x7f102c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:35256/0 (socket says 172.21.15.152:35256) 2023-12-06T04:01:34.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.201+0000 7f102a59c700 1 -- 172.21.15.152:0/655699718 learned_addr learned my addr 172.21.15.152:0/655699718 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:34.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.202+0000 7f102a59c700 1 -- 172.21.15.152:0/655699718 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f101c005040 con 0x7f102c0f5410 2023-12-06T04:01:34.204 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.202+0000 7f102a59c700 1 --2- 172.21.15.152:0/655699718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f102c0f5410 0x7f102c105630 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f101c013510 tx=0x7f101c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:34.205 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.202+0000 7f1022ffd700 1 -- 172.21.15.152:0/655699718 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f101c009320 con 0x7f102c0f5410 2023-12-06T04:01:34.205 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.202+0000 7f1030d04700 1 -- 172.21.15.152:0/655699718 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f102c105d20 con 0x7f102c0f5410 2023-12-06T04:01:34.205 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.202+0000 7f1022ffd700 1 -- 172.21.15.152:0/655699718 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f101c013f00 con 0x7f102c0f5410 2023-12-06T04:01:34.205 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.202+0000 7f1022ffd700 1 -- 172.21.15.152:0/655699718 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f101c014d90 con 0x7f102c0f5410 2023-12-06T04:01:34.205 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.202+0000 7f1030d04700 1 -- 172.21.15.152:0/655699718 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f102c1061b0 con 0x7f102c0f5410 2023-12-06T04:01:34.206 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.203+0000 7f1022ffd700 1 -- 172.21.15.152:0/655699718 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f101c014420 con 0x7f102c0f5410 2023-12-06T04:01:34.206 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.203+0000 7f1030d04700 1 -- 172.21.15.152:0/655699718 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f102c045ad0 con 0x7f102c0f5410 2023-12-06T04:01:34.206 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.203+0000 7f1022ffd700 1 --2- 172.21.15.152:0/655699718 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f101802ff60 0x7f1018032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:34.206 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.203+0000 7f1022ffd700 1 -- 172.21.15.152:0/655699718 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f101c041000 con 0x7f102c0f5410 2023-12-06T04:01:34.206 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.203+0000 7f1029d9b700 1 --2- 172.21.15.152:0/655699718 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f101802ff60 0x7f1018032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:34.206 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.204+0000 7f1029d9b700 1 --2- 172.21.15.152:0/655699718 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f101802ff60 0x7f1018032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f10140096a0 tx=0x7f1014006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:34.209 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.208+0000 7f1022ffd700 1 -- 172.21.15.152:0/655699718 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f101c0095f0 con 0x7f102c0f5410 2023-12-06T04:01:34.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.420+0000 7f1030d04700 1 -- 172.21.15.152:0/655699718 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f102c0fae60 con 0x7f102c0f5410 2023-12-06T04:01:34.422 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.421+0000 7f1022ffd700 1 -- 172.21.15.152:0/655699718 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f101c01db70 con 0x7f102c0f5410 2023-12-06T04:01:34.422 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:34.422 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.422+0000 7f1030d04700 1 -- 172.21.15.152:0/655699718 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f101802ff60 msgr2=0x7f1018032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.422+0000 7f1030d04700 1 --2- 172.21.15.152:0/655699718 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f101802ff60 0x7f1018032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f10140096a0 tx=0x7f1014006b40).stop 2023-12-06T04:01:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.422+0000 7f1030d04700 1 -- 172.21.15.152:0/655699718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f102c0f5410 msgr2=0x7f102c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.422+0000 7f1030d04700 1 --2- 172.21.15.152:0/655699718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f102c0f5410 0x7f102c105630 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f101c013510 tx=0x7f101c00b040).stop 2023-12-06T04:01:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.422+0000 7f1030d04700 1 -- 172.21.15.152:0/655699718 shutdown_connections 2023-12-06T04:01:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.422+0000 7f1030d04700 1 --2- 172.21.15.152:0/655699718 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f101802ff60 0x7f1018032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:34.424 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.422+0000 7f1030d04700 1 --2- 172.21.15.152:0/655699718 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f102c0f5410 0x7f102c105630 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:34.425 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.422+0000 7f1030d04700 1 -- 172.21.15.152:0/655699718 >> 172.21.15.152:0/655699718 conn(0x7f102c0f0b70 msgr2=0x7f102c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:34.425 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.422+0000 7f1030d04700 1 -- 172.21.15.152:0/655699718 shutdown_connections 2023-12-06T04:01:34.425 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:34.422+0000 7f1030d04700 1 -- 172.21.15.152:0/655699718 wait complete. 2023-12-06T04:01:34.425 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:35.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:34 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:34.422155+0000 mon.smithi132 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.152:0/655699718' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:35.792 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:35.792 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:36.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:35 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:34.468676+0000 mgr.smithi132.gokxja (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:36.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.890+0000 7efe32c1f700 1 -- 172.21.15.152:0/3350885581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe2c0f6510 msgr2=0x7efe2c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:36.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.890+0000 7efe32c1f700 1 --2- 172.21.15.152:0/3350885581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe2c0f6510 0x7efe2c0f6930 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7efe20009fb0 tx=0x7efe20009b30).stop 2023-12-06T04:01:36.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.890+0000 7efe32c1f700 1 -- 172.21.15.152:0/3350885581 shutdown_connections 2023-12-06T04:01:36.893 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.890+0000 7efe32c1f700 1 --2- 172.21.15.152:0/3350885581 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe2c0f6510 0x7efe2c0f6930 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:36.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.890+0000 7efe32c1f700 1 -- 172.21.15.152:0/3350885581 >> 172.21.15.152:0/3350885581 conn(0x7efe2c0f1c70 msgr2=0x7efe2c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:36.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.890+0000 7efe32c1f700 1 -- 172.21.15.152:0/3350885581 shutdown_connections 2023-12-06T04:01:36.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.890+0000 7efe32c1f700 1 -- 172.21.15.152:0/3350885581 wait complete. 2023-12-06T04:01:36.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.890+0000 7efe32c1f700 1 Processor -- start 2023-12-06T04:01:36.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.890+0000 7efe32c1f700 1 -- start start 2023-12-06T04:01:36.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.891+0000 7efe32c1f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe2c0f6510 0x7efe2c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:36.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.891+0000 7efe32c1f700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe2c069e20 con 0x7efe2c0f6510 2023-12-06T04:01:36.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.891+0000 7efe309bb700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe2c0f6510 0x7efe2c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:36.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.891+0000 7efe309bb700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe2c0f6510 0x7efe2c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:35266/0 (socket says 172.21.15.152:35266) 2023-12-06T04:01:36.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.891+0000 7efe309bb700 1 -- 172.21.15.152:0/2250079311 learned_addr learned my addr 172.21.15.152:0/2250079311 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:36.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.891+0000 7efe309bb700 1 -- 172.21.15.152:0/2250079311 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe20005040 con 0x7efe2c0f6510 2023-12-06T04:01:36.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.891+0000 7efe309bb700 1 --2- 172.21.15.152:0/2250079311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe2c0f6510 0x7efe2c06b9a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7efe20013510 tx=0x7efe2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:36.894 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.891+0000 7efe297fa700 1 -- 172.21.15.152:0/2250079311 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe20009320 con 0x7efe2c0f6510 2023-12-06T04:01:36.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.891+0000 7efe32c1f700 1 -- 172.21.15.152:0/2250079311 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe2c069fc0 con 0x7efe2c0f6510 2023-12-06T04:01:36.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.891+0000 7efe297fa700 1 -- 172.21.15.152:0/2250079311 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efe20013f00 con 0x7efe2c0f6510 2023-12-06T04:01:36.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.891+0000 7efe297fa700 1 -- 172.21.15.152:0/2250079311 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe20014d90 con 0x7efe2c0f6510 2023-12-06T04:01:36.895 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.892+0000 7efe32c1f700 1 -- 172.21.15.152:0/2250079311 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe2c06a450 con 0x7efe2c0f6510 2023-12-06T04:01:36.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.892+0000 7efe297fa700 1 -- 172.21.15.152:0/2250079311 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efe200263a0 con 0x7efe2c0f6510 2023-12-06T04:01:36.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.892+0000 7efe297fa700 1 --2- 172.21.15.152:0/2250079311 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7efe1c02ff60 0x7efe1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:36.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.892+0000 7efe297fa700 1 -- 172.21.15.152:0/2250079311 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efe20041310 con 0x7efe2c0f6510 2023-12-06T04:01:36.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.893+0000 7efe32c1f700 1 -- 172.21.15.152:0/2250079311 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe2c059100 con 0x7efe2c0f6510 2023-12-06T04:01:36.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.893+0000 7efe2bfff700 1 --2- 172.21.15.152:0/2250079311 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7efe1c02ff60 0x7efe1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:36.896 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.893+0000 7efe2bfff700 1 --2- 172.21.15.152:0/2250079311 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7efe1c02ff60 0x7efe1c032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7efe180096a0 tx=0x7efe18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:36.899 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:36.898+0000 7efe297fa700 1 -- 172.21.15.152:0/2250079311 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efe20012920 con 0x7efe2c0f6510 2023-12-06T04:01:37.109 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:37.107+0000 7efe32c1f700 1 -- 172.21.15.152:0/2250079311 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe2c06ab50 con 0x7efe2c0f6510 2023-12-06T04:01:37.109 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:37.107+0000 7efe297fa700 1 -- 172.21.15.152:0/2250079311 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efe2002b030 con 0x7efe2c0f6510 2023-12-06T04:01:37.109 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:37.109 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:37.111 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:37.109+0000 7efe32c1f700 1 -- 172.21.15.152:0/2250079311 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7efe1c02ff60 msgr2=0x7efe1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:37.111 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:37.109+0000 7efe32c1f700 1 --2- 172.21.15.152:0/2250079311 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7efe1c02ff60 0x7efe1c032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7efe180096a0 tx=0x7efe18006b40).stop 2023-12-06T04:01:37.111 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:37.109+0000 7efe32c1f700 1 -- 172.21.15.152:0/2250079311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe2c0f6510 msgr2=0x7efe2c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:37.111 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:37.109+0000 7efe32c1f700 1 --2- 172.21.15.152:0/2250079311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe2c0f6510 0x7efe2c06b9a0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7efe20013510 tx=0x7efe2000b040).stop 2023-12-06T04:01:37.111 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:37.109+0000 7efe32c1f700 1 -- 172.21.15.152:0/2250079311 shutdown_connections 2023-12-06T04:01:37.111 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:37.109+0000 7efe32c1f700 1 --2- 172.21.15.152:0/2250079311 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7efe1c02ff60 0x7efe1c032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:37.111 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:37.109+0000 7efe32c1f700 1 --2- 172.21.15.152:0/2250079311 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7efe2c0f6510 0x7efe2c06b9a0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:37.111 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:37.109+0000 7efe32c1f700 1 -- 172.21.15.152:0/2250079311 >> 172.21.15.152:0/2250079311 conn(0x7efe2c0f1c70 msgr2=0x7efe2c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:37.111 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:37.109+0000 7efe32c1f700 1 -- 172.21.15.152:0/2250079311 shutdown_connections 2023-12-06T04:01:37.112 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:37.109+0000 7efe32c1f700 1 -- 172.21.15.152:0/2250079311 wait complete. 2023-12-06T04:01:37.112 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:38.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:36.468993+0000 mgr.smithi132.gokxja (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:38.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:37 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:37.109028+0000 mon.smithi132 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.152:0/2250079311' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:38.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:38.475 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:39.561 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.559+0000 7f5a96175700 1 -- 172.21.15.152:0/481269719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a900f51a0 msgr2=0x7f5a900f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:39.561 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.559+0000 7f5a96175700 1 --2- 172.21.15.152:0/481269719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a900f51a0 0x7f5a900f55c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f5a80009fb0 tx=0x7f5a80009b30).stop 2023-12-06T04:01:39.561 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.559+0000 7f5a96175700 1 -- 172.21.15.152:0/481269719 shutdown_connections 2023-12-06T04:01:39.561 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.559+0000 7f5a96175700 1 --2- 172.21.15.152:0/481269719 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a900f51a0 0x7f5a900f55c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:39.561 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.559+0000 7f5a96175700 1 -- 172.21.15.152:0/481269719 >> 172.21.15.152:0/481269719 conn(0x7f5a900f09a0 msgr2=0x7f5a900f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:39.561 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.559+0000 7f5a96175700 1 -- 172.21.15.152:0/481269719 shutdown_connections 2023-12-06T04:01:39.561 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.559+0000 7f5a96175700 1 -- 172.21.15.152:0/481269719 wait complete. 2023-12-06T04:01:39.564 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.559+0000 7f5a96175700 1 Processor -- start 2023-12-06T04:01:39.564 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.559+0000 7f5a96175700 1 -- start start 2023-12-06T04:01:39.564 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.560+0000 7f5a96175700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a900f51a0 0x7f5a901091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:39.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.560+0000 7f5a96175700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a90060220 con 0x7f5a900f51a0 2023-12-06T04:01:39.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.560+0000 7f5a95173700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a900f51a0 0x7f5a901091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:39.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.560+0000 7f5a95173700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a900f51a0 0x7f5a901091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:47886/0 (socket says 172.21.15.152:47886) 2023-12-06T04:01:39.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.560+0000 7f5a95173700 1 -- 172.21.15.152:0/674500126 learned_addr learned my addr 172.21.15.152:0/674500126 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:39.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.560+0000 7f5a95173700 1 -- 172.21.15.152:0/674500126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a80005040 con 0x7f5a900f51a0 2023-12-06T04:01:39.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.560+0000 7f5a95173700 1 --2- 172.21.15.152:0/674500126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a900f51a0 0x7f5a901091d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f5a80013510 tx=0x7f5a8000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:39.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.560+0000 7f5a85ffb700 1 -- 172.21.15.152:0/674500126 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a80009320 con 0x7f5a900f51a0 2023-12-06T04:01:39.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.561+0000 7f5a96175700 1 -- 172.21.15.152:0/674500126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a90060420 con 0x7f5a900f51a0 2023-12-06T04:01:39.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.561+0000 7f5a85ffb700 1 -- 172.21.15.152:0/674500126 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5a80013f00 con 0x7f5a900f51a0 2023-12-06T04:01:39.565 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.561+0000 7f5a85ffb700 1 -- 172.21.15.152:0/674500126 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a80014dc0 con 0x7f5a900f51a0 2023-12-06T04:01:39.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.561+0000 7f5a96175700 1 -- 172.21.15.152:0/674500126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a90109880 con 0x7f5a900f51a0 2023-12-06T04:01:39.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.562+0000 7f5a85ffb700 1 -- 172.21.15.152:0/674500126 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5a800095f0 con 0x7f5a900f51a0 2023-12-06T04:01:39.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.562+0000 7f5a96175700 1 -- 172.21.15.152:0/674500126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a90059100 con 0x7f5a900f51a0 2023-12-06T04:01:39.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.562+0000 7f5a85ffb700 1 --2- 172.21.15.152:0/674500126 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5a7c02ff60 0x7f5a7c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:39.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.562+0000 7f5a85ffb700 1 -- 172.21.15.152:0/674500126 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5a80041020 con 0x7f5a900f51a0 2023-12-06T04:01:39.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.562+0000 7f5a94972700 1 --2- 172.21.15.152:0/674500126 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5a7c02ff60 0x7f5a7c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:39.566 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.562+0000 7f5a94972700 1 --2- 172.21.15.152:0/674500126 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5a7c02ff60 0x7f5a7c032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5a8c0096a0 tx=0x7f5a8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:39.568 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.566+0000 7f5a85ffb700 1 -- 172.21.15.152:0/674500126 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5a80014420 con 0x7f5a900f51a0 2023-12-06T04:01:39.775 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.772+0000 7f5a96175700 1 -- 172.21.15.152:0/674500126 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a900fabe0 con 0x7f5a900f51a0 2023-12-06T04:01:39.775 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.773+0000 7f5a85ffb700 1 -- 172.21.15.152:0/674500126 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5a8001c920 con 0x7f5a900f51a0 2023-12-06T04:01:39.776 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:39.776 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:39.778 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.775+0000 7f5a96175700 1 -- 172.21.15.152:0/674500126 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5a7c02ff60 msgr2=0x7f5a7c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:39.778 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.775+0000 7f5a96175700 1 --2- 172.21.15.152:0/674500126 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5a7c02ff60 0x7f5a7c032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5a8c0096a0 tx=0x7f5a8c006b40).stop 2023-12-06T04:01:39.778 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.775+0000 7f5a96175700 1 -- 172.21.15.152:0/674500126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a900f51a0 msgr2=0x7f5a901091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:39.778 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.775+0000 7f5a96175700 1 --2- 172.21.15.152:0/674500126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a900f51a0 0x7f5a901091d0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f5a80013510 tx=0x7f5a8000b040).stop 2023-12-06T04:01:39.779 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.775+0000 7f5a96175700 1 -- 172.21.15.152:0/674500126 shutdown_connections 2023-12-06T04:01:39.779 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.775+0000 7f5a96175700 1 --2- 172.21.15.152:0/674500126 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7f5a7c02ff60 0x7f5a7c032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:39.779 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.775+0000 7f5a96175700 1 --2- 172.21.15.152:0/674500126 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7f5a900f51a0 0x7f5a901091d0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:39.779 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.775+0000 7f5a96175700 1 -- 172.21.15.152:0/674500126 >> 172.21.15.152:0/674500126 conn(0x7f5a900f09a0 msgr2=0x7f5a900f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:39.779 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.775+0000 7f5a96175700 1 -- 172.21.15.152:0/674500126 shutdown_connections 2023-12-06T04:01:39.779 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:39.775+0000 7f5a96175700 1 -- 172.21.15.152:0/674500126 wait complete. 2023-12-06T04:01:39.780 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:40.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:39 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:38.469351+0000 mgr.smithi132.gokxja (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:41.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:40 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:39.774679+0000 mon.smithi132 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.152:0/674500126' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:41.151 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-06T04:01:41.151 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 -- ceph mon dump -f json 2023-12-06T04:01:42.106 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:41 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: cluster 2023-12-06T04:01:40.469692+0000 mgr.smithi132.gokxja (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-06T04:01:42.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.231+0000 7ffbd3683700 1 -- 172.21.15.152:0/354280813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbcc060280 msgr2=0x7ffbcc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:42.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.231+0000 7ffbd3683700 1 --2- 172.21.15.152:0/354280813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbcc060280 0x7ffbcc0f90f0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7ffbbc009fb0 tx=0x7ffbbc009b30).stop 2023-12-06T04:01:42.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.231+0000 7ffbd3683700 1 -- 172.21.15.152:0/354280813 shutdown_connections 2023-12-06T04:01:42.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.231+0000 7ffbd3683700 1 --2- 172.21.15.152:0/354280813 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbcc060280 0x7ffbcc0f90f0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:42.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.231+0000 7ffbd3683700 1 -- 172.21.15.152:0/354280813 >> 172.21.15.152:0/354280813 conn(0x7ffbcc0f0b70 msgr2=0x7ffbcc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:42.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.231+0000 7ffbd3683700 1 -- 172.21.15.152:0/354280813 shutdown_connections 2023-12-06T04:01:42.235 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.231+0000 7ffbd3683700 1 -- 172.21.15.152:0/354280813 wait complete. 2023-12-06T04:01:42.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.232+0000 7ffbd3683700 1 Processor -- start 2023-12-06T04:01:42.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.232+0000 7ffbd3683700 1 -- start start 2023-12-06T04:01:42.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.232+0000 7ffbd3683700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbcc060280 0x7ffbcc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:42.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.232+0000 7ffbd3683700 1 -- --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc106bd0 con 0x7ffbcc060280 2023-12-06T04:01:42.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.232+0000 7ffbd141f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbcc060280 0x7ffbcc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:42.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.232+0000 7ffbd141f700 1 --2- >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbcc060280 0x7ffbcc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.132:3300/0 says I am v2:172.21.15.152:47906/0 (socket says 172.21.15.152:47906) 2023-12-06T04:01:42.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.232+0000 7ffbd141f700 1 -- 172.21.15.152:0/212124129 learned_addr learned my addr 172.21.15.152:0/212124129 (peer_addr_for_me v2:172.21.15.152:0/0) 2023-12-06T04:01:42.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.233+0000 7ffbd141f700 1 -- 172.21.15.152:0/212124129 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbbc005040 con 0x7ffbcc060280 2023-12-06T04:01:42.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.233+0000 7ffbd141f700 1 --2- 172.21.15.152:0/212124129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbcc060280 0x7ffbcc1066e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7ffbbc013510 tx=0x7ffbbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:42.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.233+0000 7ffbc1ffb700 1 -- 172.21.15.152:0/212124129 <== mon.0 v2:172.21.15.132:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbbc009320 con 0x7ffbcc060280 2023-12-06T04:01:42.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.233+0000 7ffbd3683700 1 -- 172.21.15.152:0/212124129 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbcc106dd0 con 0x7ffbcc060280 2023-12-06T04:01:42.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.233+0000 7ffbc1ffb700 1 -- 172.21.15.152:0/212124129 <== mon.0 v2:172.21.15.132:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffbbc013f00 con 0x7ffbcc060280 2023-12-06T04:01:42.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.233+0000 7ffbc1ffb700 1 -- 172.21.15.152:0/212124129 <== mon.0 v2:172.21.15.132:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffbbc014dc0 con 0x7ffbcc060280 2023-12-06T04:01:42.236 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.233+0000 7ffbd3683700 1 -- 172.21.15.152:0/212124129 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbcc107260 con 0x7ffbcc060280 2023-12-06T04:01:42.237 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.234+0000 7ffbc1ffb700 1 -- 172.21.15.152:0/212124129 <== mon.0 v2:172.21.15.132:3300/0 4 ==== mgrmap(e 16) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffbbc0095f0 con 0x7ffbcc060280 2023-12-06T04:01:42.237 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.234+0000 7ffbd3683700 1 -- 172.21.15.152:0/212124129 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbcc045ad0 con 0x7ffbcc060280 2023-12-06T04:01:42.238 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.234+0000 7ffbc1ffb700 1 --2- 172.21.15.152:0/212124129 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffbb802ff60 0x7ffbb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-06T04:01:42.238 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.234+0000 7ffbc1ffb700 1 -- 172.21.15.152:0/212124129 <== mon.0 v2:172.21.15.132:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffbbc0410e0 con 0x7ffbcc060280 2023-12-06T04:01:42.238 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.235+0000 7ffbd0c1e700 1 --2- 172.21.15.152:0/212124129 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffbb802ff60 0x7ffbb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-06T04:01:42.238 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.235+0000 7ffbd0c1e700 1 --2- 172.21.15.152:0/212124129 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffbb802ff60 0x7ffbb8032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7ffbc80096a0 tx=0x7ffbc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-06T04:01:42.240 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.239+0000 7ffbc1ffb700 1 -- 172.21.15.152:0/212124129 <== mon.0 v2:172.21.15.132:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffbbc014420 con 0x7ffbcc060280 2023-12-06T04:01:42.447 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.445+0000 7ffbd3683700 1 -- 172.21.15.152:0/212124129 --> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffbcc1080b0 con 0x7ffbcc060280 2023-12-06T04:01:42.448 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.446+0000 7ffbc1ffb700 1 -- 172.21.15.152:0/212124129 <== mon.0 v2:172.21.15.132:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffbbc01c920 con 0x7ffbcc060280 2023-12-06T04:01:42.448 INFO:teuthology.orchestra.run.smithi152.stdout: 2023-12-06T04:01:42.448 INFO:teuthology.orchestra.run.smithi152.stdout:{"epoch":1,"fsid":"c121c6d8-93ea-11ee-95a2-87774f69a715","modified":"2023-12-06T03:52:01.531218Z","created":"2023-12-06T03:52:01.531218Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi132","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.132:3300","nonce":0},{"type":"v1","addr":"172.21.15.132:6789","nonce":0}]},"addr":"172.21.15.132:6789/0","public_addr":"172.21.15.132:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-06T04:01:42.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.448+0000 7ffbd3683700 1 -- 172.21.15.152:0/212124129 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffbb802ff60 msgr2=0x7ffbb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:42.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.448+0000 7ffbd3683700 1 --2- 172.21.15.152:0/212124129 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffbb802ff60 0x7ffbb8032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7ffbc80096a0 tx=0x7ffbc8006b40).stop 2023-12-06T04:01:42.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.448+0000 7ffbd3683700 1 -- 172.21.15.152:0/212124129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbcc060280 msgr2=0x7ffbcc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-06T04:01:42.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.448+0000 7ffbd3683700 1 --2- 172.21.15.152:0/212124129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbcc060280 0x7ffbcc1066e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7ffbbc013510 tx=0x7ffbbc00b040).stop 2023-12-06T04:01:42.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.448+0000 7ffbd3683700 1 -- 172.21.15.152:0/212124129 shutdown_connections 2023-12-06T04:01:42.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.448+0000 7ffbd3683700 1 --2- 172.21.15.152:0/212124129 >> [v2:172.21.15.132:6800/1042262986,v1:172.21.15.132:6801/1042262986] conn(0x7ffbb802ff60 0x7ffbb8032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:42.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.448+0000 7ffbd3683700 1 --2- 172.21.15.152:0/212124129 >> [v2:172.21.15.132:3300/0,v1:172.21.15.132:6789/0] conn(0x7ffbcc060280 0x7ffbcc1066e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-06T04:01:42.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.448+0000 7ffbd3683700 1 -- 172.21.15.152:0/212124129 >> 172.21.15.152:0/212124129 conn(0x7ffbcc0f0b70 msgr2=0x7ffbcc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-06T04:01:42.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.448+0000 7ffbd3683700 1 -- 172.21.15.152:0/212124129 shutdown_connections 2023-12-06T04:01:42.450 INFO:teuthology.orchestra.run.smithi152.stderr:2023-12-06T04:01:42.448+0000 7ffbd3683700 1 -- 172.21.15.152:0/212124129 wait complete. 2023-12-06T04:01:42.451 INFO:teuthology.orchestra.run.smithi152.stderr:dumped monmap epoch 1 2023-12-06T04:01:42.843 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-12-06T04:01:42.844 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-12-06T04:01:42.844 DEBUG:teuthology.orchestra.run.smithi132:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-06T04:01:42.869 DEBUG:teuthology.orchestra.run.smithi152:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-06T04:01:42.886 INFO:tasks.cephadm:Stopping all daemons... 2023-12-06T04:01:42.887 INFO:tasks.cephadm.mon.smithi132:Stopping mon.smithi132... 2023-12-06T04:01:42.887 DEBUG:teuthology.orchestra.run.smithi132:> sudo systemctl stop ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi132 2023-12-06T04:01:42.941 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:42 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: audit 2023-12-06T04:01:42.447654+0000 mon.smithi132 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.152:0/212124129' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-06T04:01:43.314 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:42 smithi132 systemd[1]: Stopping Ceph mon.smithi132 for c121c6d8-93ea-11ee-95a2-87774f69a715... 2023-12-06T04:01:43.606 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T04:01:43.311+0000 7f5c31bd4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi132 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-06T04:01:43.607 INFO:journalctl@ceph.mon.smithi132.smithi132.stdout:Dec 06 04:01:43 smithi132 ceph-c121c6d8-93ea-11ee-95a2-87774f69a715-mon.smithi132[111697]: debug 2023-12-06T04:01:43.311+0000 7f5c31bd4700 -1 mon.smithi132@0(leader) e1 *** Got Signal Terminated *** 2023-12-06T04:01:43.756 DEBUG:teuthology.orchestra.run.smithi132:> sudo pkill -f 'journalctl -f -n 0 -u ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi132.service' 2023-12-06T04:01:43.850 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-06T04:01:43.850 INFO:tasks.cephadm.mon.smithi132:Stopped mon.smithi132 2023-12-06T04:01:43.850 INFO:tasks.cephadm.mon.smithi152:Stopping mon.smithi152... 2023-12-06T04:01:43.850 DEBUG:teuthology.orchestra.run.smithi152:> sudo systemctl stop ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi152 2023-12-06T04:01:43.888 INFO:teuthology.orchestra.run.smithi152.stderr:Failed to stop ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi152.service: Unit ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi152.service not loaded. 2023-12-06T04:01:43.890 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-12-06T04:01:43.890 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi152" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi152 with status 5: 'sudo systemctl stop ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi152' 2023-12-06T04:01:43.891 INFO:tasks.cephadm:Archiving crash dumps... 2023-12-06T04:01:43.893 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7478700/remote/ubuntu@smithi132.front.sepia.ceph.com/crash 2023-12-06T04:01:43.894 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/crash -- . 2023-12-06T04:01:43.931 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7478700/remote/ubuntu@smithi152.front.sepia.ceph.com/crash 2023-12-06T04:01:43.931 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/crash -- . 2023-12-06T04:01:43.983 INFO:teuthology.orchestra.run.smithi152.stderr:tar: /var/lib/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/crash: Cannot open: No such file or directory 2023-12-06T04:01:43.983 INFO:teuthology.orchestra.run.smithi152.stderr:tar: Error is not recoverable: exiting now 2023-12-06T04:01:43.984 INFO:tasks.cephadm:Checking cluster log for badness... 2023-12-06T04:01:43.984 DEBUG:teuthology.orchestra.run.smithi132:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-06T04:01:44.016 INFO:teuthology.orchestra.run.smithi132.stderr:grep: /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/ceph.log: No such file or directory 2023-12-06T04:01:44.018 INFO:tasks.cephadm:Compressing logs... 2023-12-06T04:01:44.018 DEBUG:teuthology.orchestra.run.smithi132:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-06T04:01:44.061 DEBUG:teuthology.orchestra.run.smithi152:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-06T04:01:44.087 INFO:teuthology.orchestra.run.smithi132.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-06T04:01:44.087 INFO:teuthology.orchestra.run.smithi132.stderr:: No such file or directory 2023-12-06T04:01:44.087 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/ceph-mgr.smithi132.gokxja.log 2023-12-06T04:01:44.087 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/ceph-volume.log 2023-12-06T04:01:44.088 INFO:teuthology.orchestra.run.smithi132.stderr:gzip -5 --verbose -- /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/ceph-mon.smithi132.log 2023-12-06T04:01:44.089 INFO:teuthology.orchestra.run.smithi132.stderr:/var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/ceph-mgr.smithi132.gokxja.log: /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/ceph-volume.log: /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/ceph-mon.smithi132.log: 90.9% -- replaced with /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/ceph-volume.log.gz 2023-12-06T04:01:44.093 INFO:teuthology.orchestra.run.smithi152.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-06T04:01:44.094 INFO:teuthology.orchestra.run.smithi152.stderr:: No such file or directory 2023-12-06T04:01:44.094 INFO:teuthology.orchestra.run.smithi132.stderr: 92.9% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-06T04:01:44.096 INFO:teuthology.orchestra.run.smithi152.stderr:/var/log/ceph/cephadm.log: 96.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-06T04:01:44.097 INFO:teuthology.orchestra.run.smithi152.stderr: 2023-12-06T04:01:44.097 INFO:teuthology.orchestra.run.smithi152.stderr:real 0m0.015s 2023-12-06T04:01:44.098 INFO:teuthology.orchestra.run.smithi152.stderr:user 0m0.013s 2023-12-06T04:01:44.098 INFO:teuthology.orchestra.run.smithi152.stderr:sys 0m0.011s 2023-12-06T04:01:44.161 INFO:teuthology.orchestra.run.smithi132.stderr: 92.7% -- replaced with /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/ceph-mgr.smithi132.gokxja.log.gz 2023-12-06T04:01:44.227 INFO:teuthology.orchestra.run.smithi132.stderr: 90.5% -- replaced with /var/log/ceph/c121c6d8-93ea-11ee-95a2-87774f69a715/ceph-mon.smithi132.log.gz 2023-12-06T04:01:44.227 INFO:teuthology.orchestra.run.smithi132.stderr: 2023-12-06T04:01:44.227 INFO:teuthology.orchestra.run.smithi132.stderr:real 0m0.151s 2023-12-06T04:01:44.228 INFO:teuthology.orchestra.run.smithi132.stderr:user 0m0.216s 2023-12-06T04:01:44.228 INFO:teuthology.orchestra.run.smithi132.stderr:sys 0m0.025s 2023-12-06T04:01:44.228 INFO:tasks.cephadm:Archiving logs... 2023-12-06T04:01:44.229 DEBUG:teuthology.misc:Transferring archived files from smithi132:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7478700/remote/ubuntu@smithi132.front.sepia.ceph.com/log 2023-12-06T04:01:44.232 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-06T04:01:44.350 DEBUG:teuthology.misc:Transferring archived files from smithi152:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7478700/remote/ubuntu@smithi152.front.sepia.ceph.com/log 2023-12-06T04:01:44.351 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-06T04:01:44.391 INFO:tasks.cephadm:Removing cluster... 2023-12-06T04:01:44.391 DEBUG:teuthology.orchestra.run.smithi132:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 --force 2023-12-06T04:02:51.310 DEBUG:teuthology.orchestra.run.smithi152:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid c121c6d8-93ea-11ee-95a2-87774f69a715 --force 2023-12-06T04:02:51.623 INFO:tasks.cephadm:Removing cephadm ... 2023-12-06T04:02:51.623 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-06T04:02:51.648 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-06T04:02:51.671 INFO:tasks.cephadm:Teardown complete 2023-12-06T04:02:51.672 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi152 with status 5: 'sudo systemctl stop ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi152' 2023-12-06T04:02:51.753 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=b28be1525f2e4f349df57db34cb36c25 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_d03fca4f35d43f61b4679936aef7bd2f01cced0b/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_f2de108beb0a6dde7f3e6c96039071828a1f6269/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi152 with status 5: 'sudo systemctl stop ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi152' 2023-12-06T04:02:51.757 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-12-06T04:02:51.794 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-06T04:02:51.946 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-06T04:02:51.947 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-06T04:02:51.963 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-06T04:02:52.046 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-06T04:02:52.060 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-06T04:02:52.060 DEBUG:teuthology.orchestra.run.smithi132:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-06T04:02:52.063 DEBUG:teuthology.orchestra.run.smithi152:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-06T04:02:52.078 INFO:teuthology.orchestra.run.smithi132.stderr:bash: ntpq: command not found 2023-12-06T04:02:52.081 INFO:teuthology.orchestra.run.smithi152.stderr:bash: ntpq: command not found 2023-12-06T04:02:52.082 INFO:teuthology.orchestra.run.smithi132.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-06T04:02:52.082 INFO:teuthology.orchestra.run.smithi132.stdout:=============================================================================== 2023-12-06T04:02:52.082 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 +438us[ +438us] +/- 82ms 2023-12-06T04:02:52.082 INFO:teuthology.orchestra.run.smithi132.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -583us[ -567us] +/- 61ms 2023-12-06T04:02:52.083 INFO:teuthology.orchestra.run.smithi132.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 43 +329us[ +344us] +/- 73ms 2023-12-06T04:02:52.083 INFO:teuthology.orchestra.run.smithi132.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T04:02:52.085 INFO:teuthology.orchestra.run.smithi152.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-06T04:02:52.086 INFO:teuthology.orchestra.run.smithi152.stdout:=============================================================================== 2023-12-06T04:02:52.086 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 +427us[ +420us] +/- 82ms 2023-12-06T04:02:52.086 INFO:teuthology.orchestra.run.smithi152.stdout:^* hv02.front.sepia.ceph.com 3 6 377 40 -584us[ -590us] +/- 61ms 2023-12-06T04:02:52.086 INFO:teuthology.orchestra.run.smithi152.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 +328us[ +328us] +/- 74ms 2023-12-06T04:02:52.086 INFO:teuthology.orchestra.run.smithi152.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-06T04:02:52.087 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-06T04:02:52.138 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-06T04:02:52.138 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-06T04:02:52.213 DEBUG:teuthology.orchestra.run.smithi132:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-06T04:02:52.301 DEBUG:teuthology.orchestra.run.smithi152:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-06T04:02:52.393 DEBUG:teuthology.orchestra.run.smithi132:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-06T04:02:52.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T04:02:52.425 DEBUG:teuthology.orchestra.run.smithi152:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-06T04:02:52.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T04:02:52.458 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-06T04:02:52.469 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-06T04:02:52.496 INFO:teuthology.task.internal:Duration was 1331.870566 seconds 2023-12-06T04:02:52.497 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-06T04:02:52.538 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-06T04:02:52.538 DEBUG:teuthology.orchestra.run.smithi132:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-06T04:02:52.540 DEBUG:teuthology.orchestra.run.smithi152:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-06T04:02:52.585 INFO:teuthology.orchestra.run.smithi132.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-06T04:02:52.586 INFO:teuthology.orchestra.run.smithi152.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-06T04:02:52.869 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-06T04:02:52.869 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi132.front.sepia.ceph.com 2023-12-06T04:02:52.870 DEBUG:teuthology.orchestra.run.smithi132:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-06T04:02:52.892 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi152.front.sepia.ceph.com 2023-12-06T04:02:52.893 DEBUG:teuthology.orchestra.run.smithi152:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-06T04:02:52.916 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-06T04:02:52.916 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-06T04:02:52.935 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-06T04:02:53.002 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-06T04:02:53.003 DEBUG:teuthology.orchestra.run.smithi132:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-06T04:02:53.018 DEBUG:teuthology.orchestra.run.smithi152:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-06T04:02:53.106 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-06T04:02:53.116 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-06T04:02:53.117 DEBUG:teuthology.orchestra.run.smithi132:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-06T04:02:53.144 DEBUG:teuthology.orchestra.run.smithi152:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-06T04:02:53.171 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-06T04:02:53.195 DEBUG:teuthology.orchestra.run.smithi132:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-06T04:02:53.198 DEBUG:teuthology.orchestra.run.smithi152:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-06T04:02:53.222 INFO:teuthology.orchestra.run.smithi132.stdout:kernel.core_pattern = core 2023-12-06T04:02:53.238 INFO:teuthology.orchestra.run.smithi152.stdout:kernel.core_pattern = core 2023-12-06T04:02:53.280 DEBUG:teuthology.orchestra.run.smithi132:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-06T04:02:53.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T04:02:53.306 DEBUG:teuthology.orchestra.run.smithi152:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-06T04:02:53.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-06T04:02:53.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-06T04:02:53.348 INFO:teuthology.task.internal:Transferring archived files... 2023-12-06T04:02:53.349 DEBUG:teuthology.misc:Transferring archived files from smithi132:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7478700/remote/smithi132 2023-12-06T04:02:53.350 DEBUG:teuthology.orchestra.run.smithi132:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-06T04:02:53.390 DEBUG:teuthology.misc:Transferring archived files from smithi152:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7478700/remote/smithi152 2023-12-06T04:02:53.391 DEBUG:teuthology.orchestra.run.smithi152:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-06T04:02:53.430 INFO:teuthology.task.internal:Removing archive directory... 2023-12-06T04:02:53.430 DEBUG:teuthology.orchestra.run.smithi132:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-06T04:02:53.433 DEBUG:teuthology.orchestra.run.smithi152:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-06T04:02:53.481 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-06T04:02:53.492 INFO:teuthology.task.internal:Not uploading archives. 2023-12-06T04:02:53.493 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-06T04:02:53.539 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-06T04:02:53.539 DEBUG:teuthology.orchestra.run.smithi132:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-06T04:02:53.541 DEBUG:teuthology.orchestra.run.smithi152:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-06T04:02:53.556 INFO:teuthology.orchestra.run.smithi132.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 6 04:02 /home/ubuntu/cephtest 2023-12-06T04:02:53.558 INFO:teuthology.orchestra.run.smithi152.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 6 04:02 /home/ubuntu/cephtest 2023-12-06T04:02:53.560 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-06T04:02:53.581 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-06T04:02:53.676 INFO:teuthology.nuke:Checking targets against current locks 2023-12-06T04:02:53.702 DEBUG:teuthology.nuke:shortname: smithi132 2023-12-06T04:02:53.702 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-06T04:02:53.727 DEBUG:teuthology.nuke:shortname: smithi152 2023-12-06T04:02:53.727 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-06T04:02:53.731 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi132.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7478700', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-06 03:31:40.837514', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK1U4aUxfPIY9Ie8QujEKrtiRdKt/HGMndDrVZPcL6Lv/9HtaQ8oih0BQ9hW/0fg6yo1+Ob9qAed8A6b96J8GVo='} 2023-12-06T04:02:53.755 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi152.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-05_19:00:15-rados-wip-yuri6-testing-2023-12-05-0753-pacific-distro-default-smithi/7478700', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-06 03:31:40.836033', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBmZf2ZACUh5sE2dzDfab8lzB22ec/8jLW/QzpPhSqZiwXTDE8pNZZN0xVpKPTSIrK472EVbbw1FYTytwYTp4+M='} 2023-12-06T04:02:53.783 INFO:teuthology.orchestra.console.smithi132:Power off 2023-12-06T04:02:53.784 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-06T04:02:53.809 DEBUG:teuthology.orchestra.console.smithi132:power off output: Chassis Power Control: Down/Off 2023-12-06T04:02:53.809 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T04:02:53.826 INFO:teuthology.orchestra.console.smithi152:Power off 2023-12-06T04:02:53.826 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-06T04:02:53.836 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2023-12-06T04:02:53.850 DEBUG:teuthology.orchestra.console.smithi152:power off output: Chassis Power Control: Down/Off 2023-12-06T04:02:53.851 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T04:02:53.876 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2023-12-06T04:02:57.838 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T04:02:57.878 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T04:02:57.949 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is on 2023-12-06T04:02:57.989 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is on 2023-12-06T04:03:01.954 DEBUG:teuthology.orchestra.console.smithi132:pexpect command: ipmitool -H smithi132.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T04:03:01.990 DEBUG:teuthology.orchestra.console.smithi152:pexpect command: ipmitool -H smithi152.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-06T04:03:02.067 DEBUG:teuthology.orchestra.console.smithi132:check power output: Chassis Power is off 2023-12-06T04:03:02.100 DEBUG:teuthology.orchestra.console.smithi152:check power output: Chassis Power is off 2023-12-06T04:03:02.168 INFO:teuthology.orchestra.console.smithi132:Power off completed 2023-12-06T04:03:02.201 INFO:teuthology.orchestra.console.smithi152:Power off completed 2023-12-06T04:03:02.306 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1331.870566368103 failure_reason: 'Command failed on smithi152 with status 5: ''sudo systemctl stop ceph-c121c6d8-93ea-11ee-95a2-87774f69a715@mon.smithi152''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=b28be1525f2e4f349df57db34cb36c25 status: fail success: false 2023-12-06T04:03:02.307 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-06T04:03:02.389 INFO:teuthology.run:FAIL